Greg Pomerantz
33ed5d9fe6
The mock can now work several trips at once: a trip picker in the topbar, per-trip state in localStorage (edits + version history survive switching), and a per-trip router key so each trip routes on its own OSRM extract. Colombia (Cartagena -> Santa Marta, 8 days) becomes the default working trip, inside a new walking-profile OSRM instance: - scripts/crop_pbf.py: streaming 3-pass PBF cropper (no PBF library; the country file is too big for osmium extract -s complete_ways on this box). Framing validated against osmium test fixtures and osrm-extract. Drops relations (foot profile ignores restrictions). - scripts/setup-osrm-colombia.sh: download -> crop (~4.6 MB) -> extract/partition/customize with the official foot.lua -> serve :5003. Verified: Getsemaní -> Castillo San Felipe 7.98 km / 96 min. - server.js: ROUTERS map, ?router=colombia|northeast on /route, /table, /router-status (per-router probe points). - app.js: legs touching a transit stop are fixed (flight/taxi/bus via M.multimodal), never re-routed; distance-0 router answers (both points on one node) read as adjacent stops, not out-of-coverage. - multi-hotel fix: day-scope map fit uses the stays covering THAT day, so a two-city trip no longer fits the whole country per day. Also: pre-cache-tiles.sh (Cartagena+Santa Marta z14-16, 2560 tiles), tile cache write fix (Node rejects flags:'x'; buffer-then-write), favicon, renderHotelMks load-order guard, tests updated for the new default trip (all four puppeteer suites green, zero console errors).
232 lines
11 KiB
Markdown
232 lines
11 KiB
Markdown
# maps/router — route-aware planning primitives
|
||
|
||
The Go backend for the "search along the route" capability
|
||
([DESIGN.md](../DESIGN.md), "Driving trips" section). Everything the LLM
|
||
would otherwise *guess* about a trip — route shape, travel times, detour
|
||
costs — is computed here and returned with provenance.
|
||
|
||
## What's in here
|
||
|
||
```
|
||
internal/geo/ dependency-free geodesy (haversine, point-to-line)
|
||
internal/route/ Router interface + backends
|
||
- valhalla.go: hosted/self-hosted Valhalla (works today;
|
||
no geometry on the hosted instance → chord fallback)
|
||
- osrm.go: local OSRM (full geometry; scripts/setup-osrm.sh)
|
||
internal/plan/ the primitives:
|
||
- stopcost.go: stop_cost = route(A,C via B) − route(A,C)
|
||
- optimize.go: optimize_stops (exhaustive ≤10 stops, greedy above)
|
||
- corridor.go: corridor = buffer band around the route
|
||
cmd/routectl/ manual CLI: route | stopcost | optimize
|
||
internal/traffic/ live-traffic pipeline (511NY → Overpass match → speeds)
|
||
cmd/traffic/ fetch | match | apply (segment-speed CSV → re-customize)
|
||
cmd/bench/ benchmark runner (bench/tasks.json)
|
||
bench/tasks.json 5 real NE-corridor tasks with recorded goldens
|
||
scripts/setup-osrm.sh self-host OSRM over the NH+MA+CT+NY extract
|
||
```
|
||
|
||
Zero third-party Go dependencies (stdlib only).
|
||
|
||
## Try it (hosted Valhalla, no setup)
|
||
|
||
```sh
|
||
go run ./cmd/routectl route --from 44.054,-71.650 --to 40.729,-73.966
|
||
# Lincoln NH → Queens NY: ~5h32m, 531 km, I-93/90/495/290 (NOT via Boston)
|
||
|
||
go run ./cmd/routectl stopcost --from 44.054,-71.650 --to 40.729,-73.966 \
|
||
--stop 42.2767,-71.806:60:cascade \
|
||
--stop 43.507,-71.548:50:ladd \
|
||
--stop 43.642,-71.156:120:mtmajor
|
||
# cascade detour= 6 min (on the I-495 corridor)
|
||
# ladd detour=17 min
|
||
# mtmajor detour=80 min (off-corridor)
|
||
|
||
go run ./cmd/routectl optimize --from 44.054,-71.650 --to 40.729,-73.966 --k 2 \
|
||
--stop 42.2767,-71.806:60:cascade --stop 43.507,-71.548:50:ladd \
|
||
--stop 43.902,-71.64:60:plymouth --stop 42.425,-71.68:60:brewsters
|
||
# order: ladd -> cascade detour: 23 min
|
||
```
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
go test ./... # unit tests, offline, fast
|
||
go test -tags integration ./... # live router (Valhalla by default)
|
||
ROUTER_BACKEND=osrm ROUTER_URL=http://localhost:5000 \
|
||
go test -tags integration ./... # against local OSRM
|
||
```
|
||
|
||
The integration tests encode the key regression: the fastest
|
||
Lincoln→Queens drive stays ≥15 km from Boston (it goes via Worcester,
|
||
I-90/I-495/I-290). That is exactly the fact an LLM confidently gets wrong.
|
||
|
||
## Benchmark
|
||
|
||
```sh
|
||
go run ./cmd/bench # valhalla goldens (bench/tasks.json)
|
||
go run ./cmd/bench --tasks bench/tasks.osrm.json # local-OSRM goldens
|
||
go run ./cmd/bench --record # re-record into the file you're running
|
||
```
|
||
|
||
Tasks are real OD pairs across NH/MA/CT/NY with real POI candidates and
|
||
dwell times. **Goldens are per backend+profile** (see profile deltas
|
||
below): `bench/tasks.json` carries Valhalla goldens, `bench/tasks.osrm.json`
|
||
local-OSRM goldens. Both currently **26 PASS / 0 FAIL**, and both
|
||
routers pick the *same* optimized orders for all 5 tasks despite their
|
||
different clocks. Check types:
|
||
|
||
| type | meaning |
|
||
|---|---|
|
||
| `golden_route_min` | direct route time within a recorded band (±10%) |
|
||
| `route_avoids` | route stays ≥ N km from a point (geometry, or bbox when the backend has none) |
|
||
| `stop_free` / `stop_detour_at_least` / `stop_detour_band` | computed detour of a candidate vs an expected band |
|
||
| `optimize_order` | the exhaustive optimum matches the recorded order |
|
||
| `crosscheck` | single multi-waypoint route ≈ matrix-summed prediction (≤ maxPct drift) |
|
||
|
||
Current state: **26 PASS / 0 FAIL** on both hosted Valhalla and local
|
||
OSRM (separate golden files).
|
||
|
||
## Self-hosted OSRM (full geometry, exact corridors) — verified
|
||
|
||
`scripts/setup-osrm.sh` is the full, tested path (no sudo, no conan):
|
||
`deps.sh` builds bzip2, Lua 5.2, oneTBB, Boost 1.84 (b2 + hand-rolled
|
||
CMake config files — boost release tarballs ship no CMake support) and
|
||
osmium-tool; then it builds OSRM 26.4.1, merges the 4 state PBFs
|
||
(osrm-extract takes one input), extracts (~8 GB peak RAM, ~4 min),
|
||
partitions, customizes, and serves on `:5000`. Artifacts live in
|
||
`$HOME/osm-build` by default (override with `W=`).
|
||
|
||
```sh
|
||
ROUTER_BACKEND=osrm ROUTER_URL=http://localhost:5000 go test -tags integration ./...
|
||
go run ./cmd/routectl route --backend osrm --from 44.054,-71.650 --to 40.729,-73.966
|
||
# 6h50m, 514.4 km, 9068 geometry points (exact corridor available)
|
||
```
|
||
|
||
With OSRM, `Route.Geometry` is populated, so the corridor is exact
|
||
(distances to the actual routed polyline) instead of the chord fallback
|
||
used with the hosted Valhalla. This is what makes
|
||
`search_along_route`'s spatial pre-filter (PostGIS `ST_DWithin` on the
|
||
corridor polygon in the v1 design) sound.
|
||
|
||
### Profile deltas matter (measured, same OD pair)
|
||
|
||
| backend | Lincoln NH → 39-84 46th St, Sunnyside |
|
||
|---|---|
|
||
| Valhalla `auto` | 327 min, 529 km (97 km/h avg) |
|
||
| OSRM stock `car.lua` | 406 min, 511 km (76 km/h avg) |
|
||
| OSRM US-tuned speed table | 377 min, 526 km (84 km/h avg) |
|
||
|
||
All three avoid Boston and take the I-93/90/495/290 corridor — same
|
||
route *shape*, different *clocks*.
|
||
|
||
Why the clocks differ: **the times are the profile's speed assumptions,
|
||
not measurements.** OSRM's stock `car.lua` lists `motorway = 90` km/h
|
||
and applies a global `speed_reduction = 0.8` → untagged motorway miles
|
||
run at ~72 km/h (45 mph). That's a generic/European-flavored default
|
||
(90 km/h mirrors the European "unsigned road = 90" convention), and US
|
||
interstates are `maxspeed`-untagged in large stretches, so the default
|
||
rules. Re-extracting the same data with a US-tuned table
|
||
(`profiles/car-us.lua`: motorway 135, trunk 130, primary 110, ...
|
||
through the same 0.8 factor) closes two-thirds of the gap (406 → 377
|
||
min) and even changes the *route choice* (526 km now — faster
|
||
motorways make longer interstates win). Valhalla's `auto` is simply
|
||
calibrated more aggressively (97 km/h average).
|
||
|
||
Consequences: (a) goldens in the bench files are per backend+profile —
|
||
re-record (`--record`) after switching; (b) any user-facing
|
||
arrival-time math must use one backend consistently per trip and say
|
||
which; (c) "computed" provenance should carry the profile name,
|
||
because two computed numbers for the same leg can differ by 20–25%.
|
||
|
||
### Second dataset: Colombia walking router (`:5003`) — verified
|
||
|
||
`scripts/setup-osrm-colombia.sh` builds a second OSRM instance over the
|
||
**official `foot.lua` profile** (real walking speeds, ~5 km/h) for the
|
||
Colombia trip (Cartagena + Santa Marta boxes).
|
||
|
||
The country PBF (329 MB) is too big for `osmium extract -s complete_ways`
|
||
on this box (~2 GB free RAM), so `crop_pbf.py` streams it in three passes
|
||
and keeps only the two city boxes (~4.6 MB, 565k nodes / 118k ways, ~3 min).
|
||
Verified: Cartagena Getsemaní → Castillo San Felipe routes 7.98 km / 96 min
|
||
(≈ 5 km/h). `mock/server.js` maps `?router=colombia` to `:5003` and
|
||
`?router=northeast` (default) to `:5000`, so each trip in the mock is
|
||
routed on its own extract.
|
||
|
||
Notes:
|
||
- The cropper **drops relations** (the foot profile ignores turn
|
||
restrictions) and node metadata; ways touching a box are kept whole.
|
||
- OSRM v26 returns `Ok` with distance 0 for out-of-coverage point pairs
|
||
(not `NoTable`); the mock treats a zero-distance answer as "adjacent
|
||
stops" and a no-route answer as fallback-to-estimate.
|
||
- `crop_pbf.py` is a from-scratch PBF reader/writer (no PBF library
|
||
available); framing = `[uint32 BE BlobHeader len][BlobHeader][Blob]`,
|
||
validated against osmium's own test fixtures and `osrm-extract`.
|
||
|
||
## Live traffic (511NY → OSRM segment speeds) — verified
|
||
|
||
OSRM v26 has a first-class mechanism for this: `osrm-customize
|
||
--segment-speed-file speeds.csv` (CSV = `nodeA,nodeB,km/h`, OSM node
|
||
pairs) rewrites edge weights and re-customizes the MLD in **~15 s — no
|
||
re-extract**. The traffic layer is a renamed copy of the dataset
|
||
(customize writes the contract files in place; v26 has no
|
||
`--output-prefix`), served by its own `osrm-routed` process.
|
||
|
||
`internal/traffic` + `cmd/traffic` implement the whole loop:
|
||
|
||
```sh
|
||
# 1. pull incidents (needs a 511NY key — 511ny.org → Sign Up →
|
||
# account → API key; the key can't be requested programmatically)
|
||
go run ./cmd/traffic fetch --key "$FIFTYONE_NY_KEY" --out events.json --raw raw.json
|
||
|
||
# 2. match each incident to OSM node pairs (Overpass: ways around the
|
||
# point; within 150 m the highest-class road wins, so "I-90" beats a
|
||
# closer service road) and expand to the affected stretch
|
||
go run ./cmd/traffic match --events events.json --out speeds.csv
|
||
|
||
# 3. copy the base dataset (CoW) + re-customize the traffic layer
|
||
go run ./cmd/traffic apply --csv speeds.csv --data $HOME/osm-build/data
|
||
# → serve: osrm-routed --algorithm mld --port 5002 $HOME/osm-build/data/northeast-traffic.osrm
|
||
```
|
||
|
||
Verified end-to-end: a synthetic `Road Closed` on a 0.87 km I-90
|
||
stretch routed at 38 s (static `:5000`) becomes 142 s on the traffic
|
||
layer (`:5002`) — same path, closure speed. `match` was tested against
|
||
live Overpass data (real OSM node IDs; class-window selection; unknown
|
||
event types ignored).
|
||
|
||
Notes:
|
||
- **Incident policy speeds are `assumed`** (policy map in
|
||
`internal/traffic/policy.go`), not measurements. Routes from the
|
||
traffic layer must carry provenance
|
||
`computed(traffic, as_of <fetch time>, policy=assumed)` per DESIGN.md
|
||
R4 — and the `as_of` age is a product-visible confidence signal.
|
||
- **Closures at 5 km/h are slow, not blocked.** True blocking needs the
|
||
`--turn-penalty-file` mechanism (untested) or a graph edit; 511NY
|
||
"Road Closed" events should probably map to blocking, not 5 km/h.
|
||
- **Free coverage is patchy**: 511NY (NY) + MassDOT (MA, separate
|
||
application) give incident-level data; there is no free
|
||
floating-car feed, so per-leg traffic coverage is a confidence tier,
|
||
not a boolean.
|
||
- 511NY API: base `https://511ny.org/api`, OData-v3 shape
|
||
(`{"d":[...]}`), 10 calls/min throttle. `fetch` decodes leniently and
|
||
archives the raw body for schema confirmation on first live run.
|
||
|
||
## Design notes
|
||
|
||
- **The route is computed, never narrated.** No function in this module
|
||
returns a time estimate from model knowledge; every number is router
|
||
output (`computed` provenance per DESIGN.md R4).
|
||
- **Matrix via pairwise route calls.** Valhalla's
|
||
`sources_to_targets` caps at 150 km, so the common path is
|
||
`BuildMatrix` (pairwise `/route`). OSRM's `Table` service is available
|
||
for the local backend when a single-call matrix matters.
|
||
- **optimize_stops semantics**: the user asked for *k* stops ("two
|
||
hikes"), so the objective minimizes total = detour + dwell + overhead
|
||
over ordered subsets of **exactly k**, with fallback (a) to the best
|
||
feasible smaller set when a time budget is given, then (b) to the best
|
||
exactly-k ignoring the budget, flagged `Feasible=false`.
|
||
"At most k" without a budget degenerates (zero stops always wins).
|
||
- **Stop coordinates are POI centroids.** The router snaps to the
|
||
network; a 200 m centroid error moves a 531 km route's detour by
|
||
seconds, not minutes.
|