Greg Pomerantz
d05ebfa72f
Ran the full stack (docker now available via sg docker): - postgis/postgis:16-3.4 up, both extracts imported: northeast 291,000 POIs (from ~/osm-build/data/northeast.osm.pbf), colombia 228,446 POIs - spatiald live on :5005, proxied by the mock server (/spatial/*, /spatial-status); UI badge + poi_near tool activate automatically Fixes discovered by running against real data: - osm2pgsql image: osm2pgsql/osm2pgsql does not exist on Docker Hub — use iboates/osm2pgsql, bypass its DB-probing entrypoint (--entrypoint osm2pgsql), password via PGPASSWORD (this build's --password flag forces an interactive prompt) - osm2pgsql 2.x schema: planet_osm_point/_polygon (not _node/_way), geometry in EPSG:3857 'way' column, -k hstore for opening_hours/ fee/website/addr:*, refresh_poi() dedupes relation polygons and uses hstore -> (not ->>) - compose file downgraded to v1-compatible 3.8 (host has docker-compose 1.29, no v2 plugin); OSM_DIR exported by import.sh - spatiald: proper SQL parameterization (@@i@@ tokens, body .. shifted past the filter args — the first renumber attempt was unsound), ST_Distance(geography) instead of the geometry-only ST_Distance_Sphere, 4-arg ST_DWithin for the corridor band, WKT in lng lat order, kind= accepts exact (amenity=restaurant), bare tag value (restaurant) or family (tourism) - mock server /spatial proxy: forward u.search (was dropping it) - app.js poi_near: 'points' param, semicolons must be %3B-encoded (Go url.Parse drops the tail of a value containing a raw ';') - README: osm2pgsql 2.x data notes + kind semantics
75 lines
3.1 KiB
Bash
Executable File
75 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Import an OSM PBF extract into the local PostGIS stack via osm2pgsql.
|
|
#
|
|
# ./import.sh [file.osm.pbf] [extract-name]
|
|
#
|
|
# file.osm.pbf default: $OSM_DIR/colombia.osm.pbf (OSM_DIR defaults to
|
|
# ~/trips/osm — the PBFs live outside the dev tree).
|
|
# The merged New England file lives at
|
|
# $HOME/osm-build/data/northeast.osm.pbf (setup-osrm.sh);
|
|
# any absolute path works, e.g.:
|
|
# ./import.sh ~/osm-build/data/northeast.osm.pbf northeast
|
|
# extract-name default: basename without extension
|
|
#
|
|
# The osm2pgsql container reaches the DB through the host-published port
|
|
# 5432 (see docker-compose.yml), so no shared compose network is needed.
|
|
# After the load, the poi table is refreshed (refresh_poi()) and the
|
|
# spatial_extract row updated.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")"
|
|
|
|
# docker-compose v1 (python) or v2 (plugin) — whichever the host has
|
|
DC=docker-compose
|
|
command -v docker >/dev/null && docker compose version >/dev/null 2>&1 && DC="docker compose"
|
|
|
|
# osm2pgsql image: iboates/osm2pgsql (the osm2pgsql/osm2pgsql repo image does
|
|
# not exist on Docker Hub). Its /entrypoint.sh does its own DB probing, so we
|
|
# bypass it and exec osm2pgsql directly.
|
|
IMG=${OSM2PGSQL_IMAGE:-iboates/osm2pgsql:latest}
|
|
|
|
OSM_DIR="${OSM_DIR:-$HOME/trips/osm}"
|
|
export OSM_DIR
|
|
PBF=${1:-$OSM_DIR/colombia.osm.pbf}
|
|
NAME=${2:-$(basename "${PBF%.osm.pbf}")}
|
|
ABS=$(realpath "$PBF")
|
|
[[ -f $ABS && $(stat -c%s "$ABS") -gt 100000 ]] || { echo "no such PBF: $ABS" >&2; exit 1; }
|
|
|
|
# 1. DB up (first boot runs schema.sql)
|
|
$DC up -d postgis
|
|
echo "waiting for postgis to be ready…"
|
|
for i in $(seq 1 60); do
|
|
if $DC exec -T postgis pg_isready -U trips -d trips >/dev/null 2>&1; then break; fi
|
|
sleep 2
|
|
[[ $i -eq 60 ]] && { echo "postgis never became ready" >&2; exit 1; }
|
|
done
|
|
|
|
# 2. osm2pgsql load (one-shot container, PBF bind-mounted ro), on the
|
|
# postgis compose network. Note: this osm2pgsql build's --password flag
|
|
# FORCES an interactive prompt, so the password goes via PGPASSWORD.
|
|
# --create is the 2.x default; the 1.x ops machinery is gone in 2.x.
|
|
NET=$(docker inspect trips-postgis | python3 -c \
|
|
"import json,sys; print(next(iter(json.load(sys.stdin)[0]['NetworkSettings']['Networks'])))")
|
|
docker run --rm \
|
|
--entrypoint osm2pgsql \
|
|
--network "$NET" \
|
|
-e PGPASSWORD=trips \
|
|
-v "$(dirname "$ABS"):/data:ro" \
|
|
"$IMG" "/data/$(basename "$ABS")" \
|
|
--host=postgis --port=5432 --database=trips --user=trips \
|
|
-k --slim --flat-nodes=/tmp/flat.nodes # -k: unmatched tags → hstore `tags`
|
|
|
|
# 3. refresh the queryable poi table + extract bookkeeping
|
|
$DC exec -T postgis psql -U trips -d trips -v ON_ERROR_STOP=1 <<SQL
|
|
INSERT INTO spatial_extract (name) VALUES ('${NAME}')
|
|
ON CONFLICT (name) DO NOTHING;
|
|
SELECT refresh_poi('${NAME}');
|
|
SQL
|
|
|
|
# 4. report
|
|
$DC exec -T postgis psql -U trips -d trips -c \
|
|
"SELECT name, poi_count, to_char(imported_at,'YYYY-MM-DD HH24:MI') AS imported
|
|
FROM spatial_extract ORDER BY name;"
|
|
|
|
echo "done — extract '$NAME' is queryable. Start the service:"
|
|
echo " $DC up -d spatiald # :5005, proxied by the mock server at /spatial/*"
|