Brazil (BR): migration notes

Internal record of the P5 migration. This is a dated record (2026-10-12) and is not shipped in the zip.

• Issue: datopian/postal-codes#41 (pc-28e.7.8, was deferred 2026-10-03)
• Producer: scripts/produce.py (+ test_produce.py), pipeline: producer: custom, refresh disabled
• Parity: unexplained by the repo's classifier (every difference is in a class it has no name for), signed off by nobody yet. Against the live zip's br.csv: 1,233,231 live rows, 1,232,818 fresh; 442 rows differ in all, 0.036% (413 only in live, 29 changed). All 442 are traced below.

Why a new producer, not a port of the legacy chain

The old chain (legacy_source_pipeline/brazil_postal_code_parse.py -> refine_raw_source.py -> run_qa.py -> build_postal_codes_product.py, with scrape_admin.py) was run on today's upstream inputs (2026-10-12) and does not reproduce the live file: 1,207,370 rows against 1,233,231, and 31,882 rows (2.59%) differ. Run from the saved prototype intermediates it was byte-identical to live (checked earlier in the session), so the committed code is not the code that produced the live file. Measured stage by stage:

Causes, each checked against the inputs:

• The committed parse blanks the municipality for every CEP that is only in the Mendeley list (23,505 live CEPs). combine_clean_data() moves the Mendeley/ruacep city into streetName and blanks cityName; the merge keeps the blank because combine_first treats "" as present. The live file has the municipality on all of them.
• additionalInfo CAPITAL / COUNTRYSIDE (520,637 / 712,594 rows in the saved raw.csv) is produced by nothing in the committed code or inputs; the committed parse writes it empty. It is dropped in the 13-column product, so it does not reach the live file.
• Municipality spelling. The live admin_name2 equals the IBGE Localidades API name for its admin_code2 on 1,233,205 of 1,233,231 rows. The legacy parse copies the source's spelling instead ("Santa Bárbara D'Oeste" vs "Santa Bárbara d'Oeste").
• run_qa.py calls sys.executable without importing sys, so its Frictionless step could not have run as committed.
• The old README's claim that the logic was verified SHA256-identical is true only of the saved intermediates, not of a run from upstream.

What the producer does

Rules, per CEP (full list in the produce.py docstring): source is OpenCEP's record, else the first Mendeley row; state from the UF; municipality by IBGE code (OpenCEP carries it; Mendeley rows are placed by UF + folded city name through kelvins, then the IBGE list); admin_name2 is the IBGE API name for the code; place_name is the bairro or the municipality if empty; "A/B" bairros keep "A" and the full name goes to alternative_city_name; control characters are removed from the bairro; ordered by state, municipality, place, CEP. Run in ~35 s from cached inputs.

Inputs (all fetched upstream, data/br.inputs.json records url, time, size, sha256): OpenCEP release 2.0.1 v1.zip (342,302,994 bytes, MIT; it is the only release in the repo, published 2024-07-08), the Mendeley CSV (99,290,030 bytes, CC BY 4.0, last modified 2020-01-03), kelvins municipios.csv (MIT), IBGE Localidades estados and municipios (the API gzips its answers; the producer decompresses). The IBGE DTB report and the ruacep file that the old chain read are not inputs and are removed from the tree (still in git history).

Parity against the live zip (2026-10-12)

Every other cell, including all 1,232,818 postal_code, admin_name1/code1, admin_code2 and the empty coordinate columns, equals live. max_row_drop is 0.015, the drop is 0.03%.

Informational: the public sample at postal.datahub.io/br/br.csv is unexplained against the fresh first 100 rows (75 unclassified cells, 14 float_suffix_dropped): it is the old mangled sample.

Verification (2026-10-12)

• Random 100 rows (seed 20261012), each checked against the raw OpenCEP JSON / Mendeley rows / kelvins / IBGE API by a separate script that imports nothing from the producer: 100 of 100 pass (96 OpenCEP, 4 Mendeley). A second 100 drawn only from Mendeley-placed CEPs (the path the legacy chain broke): 100 of 100 pass.
• Municipality coverage: 5,570 municipalities have a row in live and in fresh, the same set. IBGE lists 5,571: the one without a CEP in any source (Boa Esperança do Norte, MT) is not in the product, as before.
• scripts/build.py br, build.py --all --check (248 ok), pytest (1,681 passed; 34 are br). Frictionless run inside the unpacked zip (datapackage.yaml + br.csv): VALID.

Licence: the ruacep.com.br scrape

Decision for the owner; nothing here decides it. cep_data.csv (9,324 lines, 9,149 distinct CEPs) came from ruacep.com.br ("todos os direitos reservados", no reuse licence).

• Live rows that depend on it: 173 of 1,233,231 = 0.014%. They exist only because of the scrape.
• 8,951 further live rows have their CEP in the scrape too, but are reproduced identically from OpenCEP and Mendeley (0 differences in any field), so the scrape contributes nothing to them. The "~7,000 rows" in .licensing-notes.md and "roughly 7,000" in ATTRIBUTION.txt are not what the data shows.
• The producer does not use the scrape, and stays inside the 1% bound without it (-173 rows). The price is losing those 173 CEPs; the product loses no municipality.
• The scrape file and the code that read it are deleted from the tree by this change (git history keeps them). The shipped README.md ("a small subset") and ATTRIBUTION.txt ("roughly 7,000 rows") still carry the caveat; they were left as they are because the owner has not decided whether to keep, reword or drop it, and the live zip still contains the 173 rows until it is republished.

Decisions

Open questions

• Sign-off of the 442 rows (0.036%) as explained, and of dropping the 173 ruacep rows.
• Keep, reword or drop the ruacep caveat in README.md / ATTRIBUTION.txt once the 173 rows are gone from the shipped file.
• The 240 Extrema CEPs: ask SeuAliado whether an older snapshot exists, or accept the loss.
• Download time and fragility. fetch_inputs buffers the 342 MB OpenCEP zip in memory and restarts it on any failure. GitHub's release CDN was very slow on 2026-10-12 (the first download took over an hour); the old parse used curl --retry -C -. If it keeps failing on the runner, resumable download belongs in scripts/producers/common.py, not here.
• Mendeley's data stops in 2018 and OpenCEP's only release is 2024-07; refresh stays disabled, so nothing will change until someone re-runs it.