Brazil (BR) build pipeline

Self-contained — runs entirely from this repo, no fedex-agents-prototype dependency. Fixed 2026-08-19: run_qa.py imported from models import PostalRecord and from datapackage_builder import build_datapackage — modules that only exist in that other repo — and hardcoded ROOT / "countries" / "BRA" subpaths throughout. Replaced with self-relative BR_DIR paths and inlined equivalents of the required-field checks and the 12-column FedEx-style datapackage descriptor (validate_row(), build_fedex_datapackage() in run_qa.py) — reviewed field-by-field against the originals for behavioral equivalence rather than re-implemented from scratch.

Two products come out of the same underlying data:

1. The FedEx-schema 12-column intermediate (refine_raw_source.py, scrape_admin.py, run_qa.py) — a different product with different licensing rules, see fedex-agents-prototype/CLAUDE.md "Two Output Formats" if available. Not what this repo ships, but this pipeline's own intermediate step.
2. This repo's 13-column postal-codes product (build_postal_codes_product.py, publish_r2.py) — converts the FedEx-shaped intermediate above into br.csv.

The real origin: legacy_source_pipeline/

Brazil's ~1.23M-row postal dataset traces back to brazil_postal_code_parse.py, which only ever existed in the separate datopian/Fedex-dataset repo before 2026-08-19. Pulled into legacy_source_pipeline/ along with its two small archive dependencies (RELATORIO_DTB_BRASIL_DISTRITO.csv — IBGE's official territorial division report; cep_data.csv — a ~9,300-row scrape of ruacep.com.br) so this repo doesn't depend on that other repo surviving. See legacy_source_pipeline/README.md for the licensing caveat on cep_data.csv (ruacep.com.br publishes no data-reuse terms — a residual, non-blocking risk disclosed in research.json and the shipped README).

Two live-fetch reliability bugs fixed in brazil_postal_code_parse.py itself, found while re-running it self-contained on 2026-08-19 (network issues, not fedex-agents-prototype-specific):

• Mendeley Data 403s requests/urllib3 outright — Cloudflare TLS/HTTP fingerprinting, not the User-Agent string (a UA header alone did not fix it). Now shells out to curl for that one fetch.
• The 342MB OpenCEP zip has no retry/resume in requests — a single dropped connection loses the whole download (observed: broke at 234MB of 342MB). Now uses curl --retry 5 -C - (resumes from a partial file rather than restarting).

Run order (from this directory)

cd legacy_source_pipeline
python3 brazil_postal_code_parse.py    # ~1.23M rows, real network fetch — can take 20+ min
cd ..
python3 refine_raw_source.py           # reads legacy_source_pipeline/final_output.csv
python3 scrape_admin.py                # fetches IBGE Localidades API fresh
python3 run_qa.py                      # -> data/Brazil.csv (FedEx-shaped intermediate)
python3 build_postal_codes_product.py  # -> br_br/br.csv (this repo's actual product)
python3 publish_r2.py                  # dry run by default; --apply to publish

Sources

• SeuAliado/OpenCEP (MIT) — primary bulk CEP data
• Mendeley Data, "Códigos de CEP do Brasil por UF, CIDADE, BAIRRO E LOGRADOURO" by Ivan Zimmermann (CC BY 4.0) — secondary CEP data, consolidated for coverage
• IBGE Localidades API — official Brazilian government admin reference
• kelvins/municipios-brasileiros (MIT) — municipality dictionary
• ruacep.com.br scrape (no stated licence — disclosed risk, see above)

Verification

The transformation logic was verified SHA256-identical to live R2 earlier 2026-08-19, run directly against the pre-refactor scripts (before they were made self-contained): 1,233,231 rows, the one postalCode "00000" backfill row (Boa Esperança do Norte, MT) correctly dropped. That run is the source of truth for correctness.

This refactor was validated statically, not by re-running the full multi-hundred-MB live fetch a second time (the OpenCEP zip download proved flaky over this connection — see the retry fix above — and a second full end-to-end run wasn't worth the network cost given the changes are narrow):

• Every modified file passes ast.parse (syntax-valid).
• grep confirms zero remaining functional references to ROOT, countries/BRA, or the removed models/datapackage_builder imports (only explanatory comments mention them).
• validate_row() was compared field-by-field against PostalRecord's validators: the only checks it doesn't reproduce (alpha-only countryCode, alnum-after-hyphen-strip postalCode, the postalCode-"00000"-forces- additionalInfo rule) never trigger on this pipeline's actual inputs — every row's countryCode is the literal "BRA", postalCode pattern conformance is already tracked separately via CEP_PATTERN, and no row reaches validate_row() with postalCode "00000" (the backfill path builds its output dict directly, bypassing validation, identically to the pre-refactor version).
• build_fedex_datapackage() was compared field-by-field against datapackage_builder.build_datapackage(): identical schema fields, with one incidental improvement (the coordinateLevel enum now includes POSTAL_CODE/MUNICIPALITY, matching the current models.py, which the original datapackage_builder.py copy hadn't caught up to) — irrelevant to BR since coordinateLevel is always empty here.

If a fresh full-fetch verification is wanted later, re-run the pipeline above end to end and diff br_br/br.csv against R2 br/br.csv.