Published

BRA legacy source pipeline

DataHub
DataHub
0

Postal Codes Dataset for Brazil, BR including name of the city, town, or place, various administrative divisions and alternative city names.

BRA legacy source pipeline

This is the real generator of the ~1.23M-row Brazil dataset that countries/BRA/scripts/refine_raw_source.py refines into data/raw.csv. It was pulled from datopian/Fedex-dataset (postal_code/scripts/custom_parsing_scripts/ brazil_postal_code_parse.py) on 2026-08-19 and copied here verbatim, because:

  1. It only ever existed in that separate repo — it was never committed anywhere in this repo, so nothing here could reproduce Brazil from scratch without it.
  2. The root-level Brazil.csv that refine_raw_source.py reads as its SRC was a loose, untracked file that got deleted this session (per human instruction, once its contents were captured in countries/BRA/data/raw.csv). Without this folder, there would be no way to regenerate that file at all.

brazil_postal_code_parse.py's final output columns (countryCode, postalCode, postalCodeFull, stateCode, stateName, county, cityName, alternativeCityName, additionalInfo, district, cityCode) match the deleted root Brazil.csv exactly, including the always-blank county quirk that falls out of its combine_clean_data() step — this is what identified it as the real generator in the first place, not an assumption.

What it actually does

Consolidates four sources (see countries/BRA/research.json approved_sources for license verification done 2026-08-19):

  • kelvins/municipios-brasileiros (MIT) — municipality/state dictionary, fetched live.
  • RELATORIO_DTB_BRASIL_DISTRITO.csv (this folder) — IBGE's official territorial division report. Government data, cached here rather than re-fetched.
  • cep_data.csv (this folder) — ~9,300 rows scraped from ruacep.com.br. This site publishes no data-reuse licence ("todos os direitos reservados") — this is the residual licensing caveat noted in research.json and in the postal-codes product's README. Cached here, not re-scraped, so the exposure doesn't grow.
  • Mendeley Data, "Códigos de CEP do Brasil…" by Ivan Zimmermann (CC BY 4.0) — fetched live from data.mendeley.com inside the script.
  • SeuAliado/OpenCEP (MIT) — fetched live as a 342MB release ZIP inside the script.

Running it

The script's own __main__ writes final_output.csv in the working directory it's run from, in the exact column layout described above. To regenerate raw.csv from scratch:

cd countries/BRA/scripts/legacy_source_pipeline
python3 brazil_postal_code_parse.py          # writes ./final_output.csv (~1.23M rows)
cp final_output.csv ../../../../Brazil.csv   # back to project root, refine_raw_source.py's SRC
cd ../../../..
python3 countries/BRA/scripts/refine_raw_source.py
python3 countries/BRA/scripts/run_qa.py

Untouched from the original except the two local file paths this README documents (ADD_DATA_PATH, IBGE_GOV_PATH) still point at ../../archive/..., which no longer exists relative to this location — point them at this folder instead (cep_data.csv, RELATORIO_DTB_BRASIL_DISTRITO.csv) before running, or pass absolute paths. Not pre-patched, so a diff against the original stays visible.