Published

Spain (ES) build pipeline

DataHub
DataHub
0

Postal Codes Dataset for Spain, ES including name of the city, town, or place, various administrative divisions and alternative city names.

Spain (ES) build pipeline

Self-contained — runs entirely from this repo, no fedex-agents-prototype dependency. Fixed 2026-08-19: every script computed ROOT = parents[3] and hardcoded countries/ESP/... / .cache/esp/... / .cache/jobs/ESP/... subpaths, correct only in that other repo's layout. All paths are now self-relative (ES_DIR = parent of this scripts/ folder); sources/, data/, es_es/ live as siblings under datasets/es/.

Two static, hand-authored reference files committed here (not re-derivable by re-running any fetch step):

  • muni_corrections.csv — 7 manually-verified municipality corrections, each with its own evidence string (CartoCiudad geocoder query results). Read by build_dataset.py.
  • delivery_README_template.md — the buyer-facing README. package_delivery.py asserted it existed in the delivery folder but nothing wrote it there; now copied in automatically.

resolve_po_box_codes.py bootstraps from the currently-published es.csv on R2 (same bucket this pipeline publishes to — a within-repo dependency, not an external one) purely as a discovery list of which PO-box-style postal codes exist; none of that file's data is trusted as-is (its own admin_name3 is the literal string "nan" and its coordinates are demonstrably wrong for these rows — see the script's docstring). Every code is independently re-resolved. Auto-fetches via R2 credentials if the cached copy is missing.

Run order (from this directory) — note the two-pass build_dataset.py

build_dataset.py must run twice: once before resolve_po_box_codes.py (to produce the WFS-only Spain.csv baseline that script diffs against), and once after (to fold the resolved PO-box codes and fetched coordinates back in). Running resolve_po_box_codes.py before any Spain.csv exists, or skipping the second build_dataset.py pass, are both real, easy-to-hit mistakes — a prior run this same day silently overwrote the resolved PO-box table with an empty one by getting this order wrong.

python3 scrape_postal.py         # CartoCiudad INSPIRE WFS -> ../data/postal_raw.csv
python3 build_admin.py           # INE municipality register -> ../data/admin.csv
python3 build_dataset.py         # PASS 1 — WFS-only baseline, no coords/PO-box yet
python3 resolve_po_box_codes.py  # needs pass-1 Spain.csv to exist
python3 fetch_coordinates.py     # threaded, rate-limited — takes a while
python3 build_dataset.py         # PASS 2 — folds in po_box_codes.csv + coords.jsonl
python3 package_delivery.py      # copies delivery_README_template.md in, builds es.zip
python3 publish_r2.py            # dry run by default; --apply to publish

fetch_coordinates.py and build_dataset.py also need two source dumps fetched fresh (not committed, regenerable):

curl -sL -o ../sources/ES.zip https://download.geonames.org/export/zip/ES.zip
unzip -o -q ../sources/ES.zip ES.txt -d ../sources/
mv ../sources/ES.txt ../sources/geonames_ES.txt

Sources

  • IGN/CNIG CartoCiudad INSPIRE WFS — CC BY 4.0 (postal codes, coordinates)
  • INE (Instituto Nacional de Estadística) municipality register — official Spanish government source (admin reference)
  • GeoNames ES — CC BY 4.0 (locality cross-check)

See fedex-agents-prototype/CLAUDE.md's Admin Level Mapping Registry entry for ESP (if available — not required to run this pipeline) for the full admin-tier and prefix-routing detail: e.g. the 40 postal codes whose prefix legitimately disagrees with the INE province, and why Wikipedia is explicitly NOT used as the admin source here (CC BY-SA share-alike would encumber a resold derivative).

Verification (2026-08-19)

Fresh end-to-end run, entirely within this repo. scrape_postal.py: 14,305 relations, 10,854 distinct codes, 8,188 distinct municipality codes, 0 rejected. build_admin.py: 8,132 municipalities, 52 provinces, 19 communities — exact match to documented counts. Pass-1 build_dataset.py: 100% admin coverage. resolve_po_box_codes.py: 288/288 resolved — exact match to the same-day fedex-agents-prototype run. Coordinate fetch and pass-2 build in progress at time of writing; this pipeline was already verified matching R2 exactly earlier the same day (14,538/14,538 rows, only a minor alternative_city_name gap on 148 PO-box rows from independent geocoder resolution) — the path refactor changes no logic, only where files are read/written.