Canada (CA): migration notes

Internal record of the P5 migration. This is a dated record (2026-10-12) and is not shipped in the zip.

• Issue: datopian/postal-codes#41 (pc-28e.7.9)
• Producer: scripts/produce.py (+ test_produce.py), pipeline: producer: custom, refresh disabled
• Parity: unexplained by the classifier (no class for the alphabetical tie-break); 398 of 899,779 rows changed (0.044%), 861 changed cells, same postal codes both sides. Owner signed off 2026-10-12 (standing source-drift rule for the drift part, explicit decision for the tie-break). Within the 1% bound

What changed since the 2026-10-06 note

The StatCan boundary zips (www12.statcan.gc.ca) still answer 403 behind Cloudflare. The owner asked to try the agency's ArcGIS REST service instead, and it works, so the producer now fetches the polygons from there. This is a source switch, run on the owner's request; the data licensor and licence are the same (see Licence).

Sources and how they are fetched

• GeoNames CA_full.csv.zip (CC BY 4.0): fetched, one request.
• StatCan 2021 cartographic boundaries, ArcGIS REST https://geo.statcan.gc.ca/geo_wa/rest/services/2021/Cartographic_boundary_files/MapServer, layer 4 (CD, 293 features) and layer 9 (CSD, 5,161 features). Requested by object id with outSR=3347 (the shapefiles' native CRS) and no generalisation parameter (geometryPrecision, maxAllowableOffset, quantizationParameters are never sent; a test asserts it). Cached as batch files in sources/arcgis/<layer>/ (about 1.4 GB), resumable per batch from <layer>.part/.
• Never PCCF: not read, joined or compared at any point.

Stall-proofing (added after the first fetch hung for 20+ minutes on one connection): every request has a 90 s wall-clock limit (urllib's own timeout only bounds each socket read, so a trickling connection hung forever); about 4 requests in 10 return an HTML "Application Error" page with HTTP 500, so a multi-feature request is tried twice, then split in halves down to one feature, and a single feature is tried 3 times as GeoJSON, then 8 times as PBF, with jittered back-off; at most 4 requests in flight; a failed batch raises and leaves the finished batches for a resume.

PBF fallback (a deviation from "GeoJSON only", agent port choice). The service cannot serialise a few features as GeoJSON or Esri JSON at all: it answers Error performing query operation after about 7 s, every time (checked repeatedly, also on the French twin service). Examples: CD 2498 Minganie (object id 144; 9,031 polygons, 1.05 M vertices) and CD 6204 Qikiqtaaluk (id 291; 3.56 M vertices); the others are also very large coastal and Arctic features. The same features come back fine as f=pbf, so the producer decodes that (decode_pbf, about 60 lines, no protobuf dependency) and stores the result as the equivalent GeoJSON feature. Verified against the shapefile: identical vertex counts, coordinates equal to within 1 mm, symmetric difference below 1e-8 of the area. Qikiqtaaluk is not OGC-valid (ring self-intersection) in the shapefile either, so validity is not required.

Runtime with a warm cache: about 105 s (GeoNames fetch, load 1.4 GB of GeoJSON, two spatial joins, FSA test). A cold fetch of both layers took about 10 min on the happy path plus a few minutes for each hard feature; budget 15-30 min.

What the producer does

The old chain (fetch_sources.py, build_base.py, spatial_join_cd.py, spatial_join_csd.py, integrity_checker.py, package.py) and the blank-coordinates step are one produce.py, ported not reinvented. The blanking step, blank_suspect_coordinates.py, was never in this repo: it lived in an untracked working folder (ca_update/build/scripts/ of the fedex-agents-prototype checkout), so it is now committed for the first time, as part of the producer. Steps are in the module docstring: merge 770 duplicate codes by the alphabetical rule below (first name by accent/case-insensitive order; the others to alternative_city_name), CD join (snap 5 km, border straddle rule 5 km, otherwise suspect), CSD join (snap 5 km, wrong province rejected, CD reconciled with CSDUID[:4]), blank suspect coordinates (join-suspect or more than 150 km from the median of a compact FSA). Output keeps the live file's CRLF line endings and is sorted by postal code, so it no longer depends on GeoNames' row order at all (the live file follows GeoNames' own block order; keyed parity ignores row order).

Duplicate-code rule (owner decision 2026-10-12). The rows of one postal code are ordered by place_name compared without case or accents (NFKD, combining marks dropped, casefold), ties by the raw string, then latitude, then longitude (numeric, a blank last). The first row gives place_name, the kept coordinate and accuracy; the other distinct names go to alternative_city_name, pipe-separated, in the same order. A test shuffles the rows of a group (20 seeds) and asserts identical output.

Port fidelity (2026-10-12)

The ported code on the local 2026-10-05 shapefiles and the 2026-10-05 GeoNames snapshot reproduces the live ca.csv byte for byte (with the first-listed duplicate rule the live file used; the producer now uses the alphabetical rule below) (apart from CRLF, which the producer then writes too). Same counters as the 2026-10-05 build log: 81 suspect coordinates and 164 border straddles in the CD join, 899,422 rows with a CSD, 328 FSA outliers, union 334 blanked.

Boundary polygons: ArcGIS vs shapefile

Compared against the 2026-10-05 shapefile build (lcd_000b21a_e, lcsd000b21a_e) feature by feature: 293/293 CDs and 5,161/5,161 CSDs, same ids, same geometry types, identical vertex counts, all coordinates equal to within 1 mm, area ratio within 1e-9 (CD) and 6e-7 (CSD, small polygons), PRUID equal. Names equal after whitespace collapse except one: CSD 6104006, the shapefile reads Sambaa Ke and the service Sambaa K’e (the DBF byte for the apostrophe was lost by the shapefile reader, so the live file has the wrong spelling; the service's is right). Conclusion: the service serves the same cartographic generalisation as the zips.

Parity against the live zip (2026-10-12, whole file, after the alphabetical rule)

Rows only in live: 0; only in new: 0. The 334 blanked coordinates are reproduced exactly (same set).

How the 398 split (three runs): the old inputs (10-05 GeoNames + shapefiles) through the new rule give 397 changed rows; today's inputs give the same 397 plus 1. So 397 rows change because of the alphabetical rule alone, 0 because of GeoNames drift (the rule is order independent, and the group contents did not change), and 1 because of the apostrophe (X0E 1Z0, Sambaa Ke to Sambaa K’e; the old shapefile lost the byte, the service has it right).

Of the 397 rule changes, 379 only swap place_name and alternative_city_name inside groups whose rows share one coordinate. 18 groups have rows at different coordinates, so the kept point moves (latitude/longitude change); this is where all 4 Census Division and 11 Census Subdivision changes come from:

The three moves over 100 km (G0C 2Z0, S0J 2E1, V0K 2S1) are groups whose names sit far apart in GeoNames itself; neither point is known to be right, and the rule only makes the choice stable. Worth a look by the owner. All other changed rows are the plain name swap.

Verification

• Random 100 rows (seed 20261013) and 100 rows drawn from duplicate-code groups only (seed 7), checked without producer code: raw GeoNames fields and the alphabetical rule re-derived independently, admin recomputed by brute-force within on the cached GeoJSON and by DuckDB ST_Contains on the shapefile (two engines, two files): 200/200 pass (fields 100/100 each; admin 100/100 each). An earlier sample (seed 20261012, before the rule change) also passed.
• The built zip (ca.zip: ca.csv, datapackage.yaml, README.md, ATTRIBUTION.txt) unzipped and Frictionless-validated from inside the folder: valid, 899,779 rows; ca.csv hash equals the descriptor's.
• build.py ca, build.py --all --check (248 ok), parity.py ca --refresh, pytest (30 ca tests; whole suite 1,677 passed).

Decisions

Licence

Full record in .licensing-notes.md. In short: the ArcGIS service and the zips' embedded metadata both say "Open Government Licence - Canada (http://open.canada.ca/en/open-government-licence-canada)"; the 2021 Boundary Files reference guide says use is governed by the "Statistics Canada Open Licence Agreement" (https://www.statcan.gc.ca/eng/reference/licence), which allows use, reproduction, publication, distribution and sale of the Information and of Value-added Products, with the notice "Adapted from Statistics Canada, name of product, reference date. This does not constitute an endorsement by Statistics Canada of this product." The owner decided on 2026-10-12 to carry both, and README.md and ATTRIBUTION.txt now do.

Open questions

• The three groups whose kept point moves more than 100 km (G0C 2Z0, S0J 2E1, V0K 2S1): leave as is, or blank the coordinate of groups whose rows disagree by more than some distance?
• No open.canada.ca record and no readable www12 page for the 2021 boundary files; a single licence name would need Statistics Canada's confirmation.