Published

Malaysia (MY) build pipeline

DataHub
DataHub
0

Postal Codes Dataset for Malaysia, MY including name of the city, town, or place, various administrative divisions and alternative city names.

Malaysia (MY) build pipeline

Self-contained — runs entirely from this repo, no fedex-agents-prototype dependency. Fixed 2026-08-19: every script computed ROOT = parents[3] and hardcoded countries/MYS/... / .cache/MYS/... subpaths, correct only in that other repo's layout. All paths are now self-relative (MY_DIR = parent of this scripts/ folder); sources/, data/, my_my/ live as siblings under datasets/my/.

Two gaps closed that weren't fedex-specific — the pipeline never had these steps at all, it always assumed pre-fetched/pre-packaged files existed:

  • fetch_source.py — new. Downloads MCMC's postcode dictionary and the GeoNames MY export.
  • run_qa.py now builds my.zip and writes validation_report.json into the delivery folder itself — publish_r2.py requires both and nothing produced them before.

The buyer-facing README.md shipped in the delivery is a static, hand-authored document. It's committed here as delivery_README_template.md (next to this file, since data//my_my/ are gitignored/regenerable) and build_deliverable.py copies it into the delivery folder.

Run order (from this directory)

python3 fetch_source.py        # MCMC postcodes.csv + GeoNames MY.zip -> ../sources/
python3 build_postal.py
python3 build_deliverable.py
python3 run_qa.py              # also builds my.zip + validation_report.json
python3 publish_r2.py          # dry run by default; --apply to publish

Sources

  • MCMC Postcode Dataset, via data.gov.my — CC BY 4.0 (postal code, state, city)
  • GeoNames MY — CC BY 4.0 (coordinates, joined by postal code)

Verification (2026-08-19)

Fresh end-to-end run, entirely within this repo. Fetched both sources fresh. build_postal.py/build_deliverable.py produced 2,931 rows, 16/16 states — exact match. Final my.csv MD5 8819f4853269ae71df5204b7771db78d, byte-for-byte identical to the live R2 object (same MD5, same 158,936 bytes). README.md byte-identical too. datapackage.json and my.zip differ by a small amount (JSON formatting / zip container metadata, not content). Frictionless valid. Publish dry-run clean.