New Zealand (NZ): migration notes

Internal record of the P5 migration from the old scripts to the producer: the discrepancies found, their causes, and who decided what. This is a dated record, so the numbers are as of the migration and not kept current. It is not shipped in the zip.

• Issue: pc-28e.7.40 (closed 2026-10-03)
• Commits: e5cef74 (producer; worktree 65ca3d0) and 84bc512 (centroid fix; worktree 2a16487), both landed unchanged, batch 5
• Parity verdict: unexplained (unclassified coordinate cells); accepted by the user as LINZ drift plus the centroid fix
• Producer: custom, refresh disabled

Sources

• The old pipeline read LINZ through the LDS WFS, which needs a LINZ_KEY. No key exists in .env or .env.prod, and a keyless request returns 401.
• The old monthly build didn't fetch LINZ at all: it used a frozen locality reference (nz_linz_reference.csv, plus statsnz_ta_codes.csv) staged in R2 at nz/linz/, of unknown date. The producer recomputes the reference from today's addresses on every run.
• The descriptor sources keep the LDS and Stats NZ datafinder pages and add the two ArcGIS services the producer reads.

Parity against the live zip (2026-10-03)

Live reference: nz/consolidated/latest/nz.zip. Rows: 1,722 live and 1,722 fresh, none only on one side. Every column except latitude and longitude matches live exactly, row order included (postcodes, place names, regions, ISO codes, TA names and codes, alternate names).

First run (65ca3d0): 893 rows changed, 1,689 cells (latitude 833, longitude 856): coordinate_precision ×112, unclassified ×1,577.

Landed (2a16487, after the centroid fix): 938 rows changed, 1,785 cells (latitude 883, longitude 902): coordinate_precision ×92, unclassified ×1,693. The fix changed 185 rows compared with the first run, coordinates only.

Causes:

• Small moves (most rows): LINZ address edits since the frozen reference. E.g. 0114 Hikurangi −35.593753 → −35.593777. Not traced cell by cell: the old reference sits in R2 outside our own zip, which the agent wasn't allowed to read.
• Same-name localities blended (old build and first run). The old reference averaged every same-name locality nationwide. Examples: 2159 Otara in live carries Southland's Otara point (−46.626667, 168.887758) with TA Auckland, which the agent couldn't reproduce from one reference; the first run gave the mean of Auckland, Ōpōtiki and Southland (−37.096046, 174.930438); landed gives Auckland's Ōtara (−36.961775, 174.879237). 1042 Hillsborough and 0140/0149 Riverside were blends in live too; landed gives Auckland's and Whangarei's. 3886/3988 Owhiro now equal live (Ōtorohanga's Ōwhiro).
• Locality renames (macrons) and new homonyms change which rows get a LINZ centroid vs community coordinates: 0472 Pakaraka, 4273 Argyll East (−39.8672, 176.5055 → −39.837184, 176.550778) and 4542 Wanganui East now get a LINZ centroid; 2025 Manukau and 3015 Fairy Springs fall back to community coordinates. Agent's reading; not traced against the old reference.
• The fix moves some rows further from live, because live had the same blending: rows more than 0.01° from live rose from 19 to 71, e.g. 0145 Kensington, 9082 Aramoana, 4614 Normanby, Oakura.

Decisions

Transforms carried over

Folded from parse_nz.py → fetch_nz_reference.py → enrich_nz.py → integrity_checker.py (all deleted with the scripts README):

• One row per postcode: the first community row wins (place_name, region, TA, lat/long); the other distinct localities for that postcode go to alternative_city_name, pipe-separated, in source order.
• admin_code1 → ISO 3166-2:NZ via a region-name crosswalk (both "Manawatu-Wanganui"/"-Whanganui" and "Hawke's"/"Hawkes Bay" spellings).
• LINZ reference: points grouped by normalised suburb_locality (accents stripped, lowercased, non-alphanumerics dropped); dominant TA per locality; centroid rounded to 6 dp; points outside lat −48..−33 / lon 165..179 (e.g. the Chathams) ignored. The TA's region is the community file's dominant region for it.
• Region-gated join: Stats NZ TA 2025 name, 3-digit code and LINZ centroid only when the locality's TA region equals the row's region; otherwise the community TA name is standardised to its Stats NZ name and code and community coordinates stay (e.g. Auckland's vs Hurunui's "Balmoral").
• admin_name3, admin_code3 and accuracy stay empty.
• Old checks are now ProducerErrors: 4-digit postcode, known region, all 16 regions present, unique postal_code, coordinates inside NZ, every row has a TA code. No row-count assert.

Open questions

• The README Fields table still has a "Fill" column with percentages; left for the end-of-epic sweep (pc-hfz).
• Resolved (user, 2026-10-05, pc-28e.7.59): the centroid fix's drift (19 → 71 rows >0.01° from live) is accepted.
• Resolved (user, 2026-10-05, pc-28e.7.59): add an ATTRIBUTION.txt for LINZ and Stats NZ (CC BY 4.0). Tracked as pc-28e.15.

Provenance

Reconstructed 2026-10-05 from the agent transcript d33e1b7e-5027-498a-a97f-ba914ee2524e/subagents/agent-ab984f25be9928a2e.jsonl (both hand-backs), the orchestrator session d33e1b7e-… (question, user answer, relayed fix instruction, landing), the two commit messages, the bd close reason and docs/parity/p5.md.