# spatial — PostGIS source of truth

Native spatial queries for the trip planner (DESIGN.md §Data: "PostGIS
(Postgres) — import PBF via osm2pgsql; gives SQL spatial queries (radius,
nearest, bbox)"). Replaces the flat-file/Overpass approximations.

## Stack

| Piece | What it is |
|---|---|
| `docker-compose.yml` | `postgis/postgis:16-3.4` + `postgresql-16-pgvector` (DB, port 5432) + one-shot `iboates/osm2pgsql` importer + `spatiald` (query service, port 5005) |
| `schema.sql` | runs on first boot: `poi` table (named POIs as points, GIST index), `spatial_extract` bookkeeping, `refresh_poi()` |
| `20-vector.sql` | runs on first boot: `CREATE EXTENSION vector` (the `vector` package is baked into the DB image — see `db/Dockerfile`) |
| `db/Dockerfile` | `postgis/postgis:16-3.4` + `postgresql-16-pgvector` (workaround for the EOL bullseye repos — see the file) |
| `import.sh` | loads a PBF from `../osm/` via osm2pgsql, refreshes `poi` |
| `main.go` (spatiald) | JSON query service over `poi` + `poi_vec` (radius / KNN / corridor / **semantic**) |
| `embed/main.go` (embedpoi) | builds the `poi_vec` semantic index from `poi` (zembed embeddings → pgvector, HNSW) |
| `Dockerfile` | builds spatiald |

## Prerequisites

* Docker with the current user in the `docker` group (on this box:
  one-time `sudo usermod -aG docker $USER`, then re-login).
* PBF extracts in `~/trips/osm` (already present: `nh.osm.pbf`,
  `colombia.osm.pbf`, US state extracts; override with `OSM_DIR`).

## Usage

```bash
docker-compose up -d postgis           # first: runs schema.sql
./import.sh ~/trips/osm/colombia.osm.pbf colombia
./import.sh ~/osm-build/data/northeast.osm.pbf northeast   # merged NE extract
docker-compose up -d spatiald          # query service on :5005
```

The mock server (`mock/server.js`) proxies it: `GET /spatial/*` → `:5005`,
plus `GET /spatial-status` for the UI badge. When the service is up, the
chat assistant gains the `poi_near` tool automatically.

## spatiald endpoints

All return `{"count":N,"results":[{extract,osm_id,name,kind,opening_hours,
fee,website,addr_city,dist_m},…]}`. Common filters:

* `extract=nh` — one loaded extract
* `kind=amenity=restaurant` — exact kind, or `kind=restaurant` (any family)
* `name=café` — ILIKE substring
* `limit=20` (max 200)

| Endpoint | Query | SQL core |
|---|---|---|
| `GET /health` | — | extract bookkeeping |
| `GET /near` | `lat,lng,r(m; default 500)` | `ST_DWithin(geography, …, r)` + KNN ordering |
| `GET /nearest` | `lat,lng` | `ORDER BY geom <-> point` |
| `GET /corridor` | `points=lng,lat;…`, `r` | `ST_DWithin(geom::geography, ST_Buffer(line::geography, r))` |
| `GET /semantic` | `q` (free text), `limit` | cosine `<=>` over `poi_vec` (pgvector HNSW) |

`/semantic` embeds `q` with the local zembed model (`EMBED_BASE` / `EMBED_MODEL`,
defaults `http://192.168.3.7:1234/v1` + `zembed-1-Q4_K_M`) and ranks `poi_vec` by
cosine distance. Each result also carries `lat`, `lng` and `score` (1 − cosine
dist) so the UI can drop map pins. The same `extract`/`kind`/`name` filters apply.

Examples:

```bash
curl 'localhost:5005/near?lat=42.35&lng=-71.06&r=1000&kind=restaurant'
curl 'localhost:5005/nearest?lat=10.40&lng=-75.54&kind=tourism&limit=5'
curl 'localhost:5005/corridor?r=300&kind=fuel&points=-71.06,42.35;-71.07,42.36'
curl 'localhost:5005/semantic?q=medieval%20museums&limit=5'
```

## Semantic index (pgvector)

`poi_vec` mirrors `poi` for every **named POI family except `place=*`** and
stores a 2560-dim zembed embedding of `"name — kind"`. It is a *derived* table
— rebuild it any time with `embedpoi` (it is resumable: existing keys are
skipped, and it drops/recreates the table only if the embedding dimension
changed).

```bash
go run ./embed -dsn "host=localhost port=5432 user=trips password=trips dbname=trips sslmode=disable"
# restrict to one extract, or be gentler on the shared model host:
go run ./embed -extract colombia -qps 2 -batch 64
```

* Embeddings come from the shared llama.cpp host (`/v1/embeddings`), so the
  host must be up. That host swaps models in/out and 500s briefly while zembed
  is (re)loaded — `embedpoi` retries patiently (up to 20× with backoff).
* Zembed uses a `query: ` prompt prefix for search-time queries — `spatiald`
  adds it in `/semantic`; `embedpoi` stores raw `"name — kind"` passages.
* After the run it builds `poi_vec_hnsw` (`hnsw (vec vector_cosine_ops)`),
  which is what makes `/semantic` fast at full scale.
* The DB image is `postgis/postgis:16-3.4` + `postgresql-16-pgvector` (see
  `db/Dockerfile`). The stock postgis images for PG16 are bullseye-based and
  their Debian release files are expired, so the image disables
  `Check-Valid-Until` and adds the pgdg repo just to fetch `postgresql-16-pgvector`.

## Data notes

* Built for **osm2pgsql 2.x** (`iboates/osm2pgsql`): tables are
  `planet_osm_point` / `planet_osm_polygon` (not the 1.x `_node`/`_way`),
  geometry lives in a `way` column in EPSG:3857, and the import runs with
  `-k` so tags without a fixed column (`opening_hours`, `fee`, `website`,
  `addr:city`) land in the hstore `tags` column that `refresh_poi()` reads.
  `refresh_poi()` also dedupes relation-derived polygons (2.x repeats them
  per relation with negated ids).
* `poi` covers every **named** point/polygon with an `amenity`/`tourism`/
  `shop`/`leisure`/`historic`/`place` tag. Unnamed amenities (a nameless
  kiosk) are out of scope for v1.
* `kind` filter semantics: `kind=amenity=restaurant` (exact),
  `kind=restaurant` (tag value, any family), `kind=tourism` (whole family).
* Corridor `points` must URL-encode the semicolons (`%3B`) — Go's
  `url.Parse` drops the tail of a value that contains a raw `;`.
* Multiple extracts coexist in one DB, disambiguated by `poi.extract`
  (set automatically by `import.sh`).
* `lib/pq` has no `[]float32` → `vector` codec, so `/semantic` sends the
  embedding as a `[f1,f2,…]` **text** literal and casts with `::vector` in SQL.
* `poi_vec` is rebuilt by `embedpoi` (not `import.sh`); it is keyed
  `(extract, osm_type, osm_id)` to match `poi`.

## Roadmap hooks

* `bbox` queries: trivial addition (`ST_Contains(ST_MakeEnvelope,…)`).
* Rerun `embedpoi` after a new `import.sh` so `poi_vec` tracks the new `poi` rows.
* Routing-graph join: OSRM/GraphHopper geometries can be loaded into the
  same DB (`route_geom` table) for true along-route analytics instead of
  the current buffer-over-polyline.