Canada Postal Codes — Data Source & Enrichment Plan

Status: Proposed Date: 2026-07-15 Owner: Data pipeline team

1. Problem

The current Canada (ca) product is generated by scripts/custom_parsing_scripts/canada_parse.py, which parses the StatCan Open Database of Addresses (ODA). Analysis shows this source is materially incomplete for a postal-code product:

• ~5% national coverage — 45,169 unique postal codes vs Canada's ~900k.
• 5 of 13 provinces/territories only — Ontario, Nova Scotia, British Columbia, Quebec, New Brunswick. Alberta, Manitoba, Saskatchewan, PEI, Newfoundland & Labrador and the three territories carry no postal codes in ODA (the postal_code field is left blank by those data providers).
• Frozen source — ODA is pinned to the 2021001 vintage. Last year's output file is byte-for-byte identical (MD5 8a656cc4e1f62a7060227c24e58ed9ac) to a fresh regeneration. Re-running the job produces the same result; there is no fuller historical edition to recover.
• 92% duplicate postal codes — ODA is an address database (many civic addresses per code).

The live published data (/ca/geonames/latest) matches neither the current StatCan output nor a clean Geonames dump — it contains malformed zero-padded codes (e.g. 0000T0A) from an older, broken parse.

2. Source comparison

Geonames CA_full is the only complete, all-province, full-postal-code source. It becomes the source of record. Its single gap is an empty second-level administrative division (admin_name2/admin_code2 — county/municipality).

3. Goal

Produce a ca dataset that:

1. Uses Geonames CA_full as the base (900k codes, all 13 provinces/territories).
2. Enriches the empty admin_name2/admin_code2 (municipality/census-subdivision) where a trustworthy source exists.
3. Retires the StatCan-only pipeline as the primary generator.
4. Fixes the malformed postal-code padding bug in the published data.

4. Enrichment design

4.1 What StatCan can contribute

Join key is postal_code. Measured overlap between Geonames CA_full and StatCan ODA:

• Overlap: 44,964 postal codes = 5.0% of Geonames, 99.5% of StatCan.
• StatCan codes absent from Geonames: 205 (negligible).

The useful level-2 signal is not in the processed StatCan file (whose admin_name2 = data provider, admin_code2 = province id). It lives in the raw ODA columns csdname (Census Subdivision name ≈ municipality) and csduid (Census Subdivision UID). Using it requires re-parsing the raw ODA zips (~300 MB+ per province) and keeping csdname/csduid.

Scope of StatCan enrichment: fills admin_name2/admin_code2 for ~5% of rows, in 5 provinces only. It adds essentially no new postal codes.

4.2 Nationwide alternative (recommended primary enrichment)

For admin2 coverage across all of Canada, apply a postal-code → census-subdivision crosswalk (StatCan Postal Code Conversion File, PCCF, or an equivalent FSA/postal→CSD mapping) to all 900k Geonames codes. This gives one enrichment source at ~100% coverage instead of a 5% patch.

4.3 Layered strategy

1. Base: Geonames CA_full → all 12 standard fields, admin2/3 empty.
2. Primary enrichment (nationwide): PCCF/CSD crosswalk fills admin_name2/admin_code2 for all rows where a mapping exists.
3. StatCan supplement (5-province): for the 44,964 overlapping codes, backfill/validate admin_name2/admin_code2 from raw ODA csdname/csduid where PCCF is missing, and add the 205 StatCan-only codes.
4. Leave admin_name3/admin_code3, alternative_city_name empty (no reliable source).

5. Implementation steps

1. New parser scripts/custom_parsing_scripts/canada_geonames_full.py:
   • Download https://download.geonames.org/export/zip/CA_full.csv.zip.
   • Map the 12 Geonames columns to the datahub 13-column schema (append empty alternative_city_name).
   • Normalize postal_code to A1A 1A1 (fixes the 0000T0A padding bug).
   • Upload to R2 as the ca resource(s).
2. Crosswalk enrichment scripts/custom_parsing_scripts/canada_admin2_enrich.py:
   • Load PCCF (or chosen crosswalk); join on postal code; fill admin_name2/admin_code2.
   • Fall back to raw-ODA csdname/csduid for the 5-province overlap where PCCF is absent.
3. Retire the StatCan-only path: remove/disable run-monthly-canada-postal-codes.yml as the primary generator (keep the raw-ODA parse only as the supplement in step 2 if used).
4. Update datasets/ca/datapackage.yml — description, modified, bytes, and source attribution to reflect Geonames CA_full + enrichment.
5. Regenerate the 100-row sample and consolidated_stats via the existing samples/stats job.
6. Validate with scripts/custom_parsing_scripts/data_integrity_checker.py (empty/duplicate/format checks) plus coverage assertions (all 13 provinces present, ~900k unique codes).

6. Expected result

7. Risks & notes

• Geonames licensing — CA_full is under the Geonames license (CC BY). Confirm redistribution terms for the premium/full product before publishing.
• PCCF licensing/access — confirm StatCan PCCF redistribution terms; if restricted, fall back to the raw-ODA csdname supplement (5-province only) and leave admin2 empty elsewhere.
• Coordinate quality — Geonames CA_full coordinates are 100% valid and within the Canada bounding box; the small CA.zip file has a few placeholder coords (lat 90 / lon 0) and must not be used as the base.
• No StatCan history gain — do not spend effort hunting older StatCan editions; the source gap is structural, not temporal.