Runbook — Fase 1: flip lag-tolerant readers to Iceberg

Stage 1 · Fase 1. Flip the EVENTUAL (lag-tolerant) dataset-row readers from Postgres to the Iceberg lakehouse. This is an operational procedure (DB flag updates + canary runs against a live ml-runner) — no code deploy is required for the readers in scope, because their Iceberg executors already exist.

Scope — readers ready to flip

All four are EVENTUAL and already carry a complete Iceberg executor in their resolveDatasetRowSource(...).run({ postgres, iceberg }):

Explicitly NOT in Fase 1 (not servable by the read endpoint today — projection + limit only, no filter/order/keyset/row_index/id): objects.sync (needs id/row_index → Phase 4), dashboards.aggregate + pipeline.expectations (need SQL compute over data → Stage 1 compute), sdk.files.list / mediaset.item (file-backed/manifest datasets, not Iceberg-tabular). The STRICT interactive readers stay on PG automatically (their freshness gate keeps them on Postgres until caught-up + the Carbon cache lands).

Why each flip is safe

The read-router (lib/lakehouse/read-router.ts) makes a flip self-protecting:

• Freshness gate (EVENTUAL): Iceberg is used only when the dataset has a succeeded snapshot. No snapshot → serves Postgres, transparently.
• PG fallback: mode iceberg (not iceberg_only) falls back to Postgres on any Iceberg read error. Correctness is preserved through transient ml-runner blips.
• Flag cache TTL is 10s — a flip (or rollback) takes effect within ~10s.

shadow mode is a no-op at the router (Postgres stays authoritative; parity is gathered offline by the canary), so the rollout goes off → iceberg directly, with the canary as the pre-flip parity gate.

Pre-requisites

1. ML_RUNNER_URL + ML_RUNNER_TOKEN set in .env.local (the reader proxies to ml-runner POST /lakehouse/read).
2. The PG→Iceberg mirror is running (ENABLE_ICEBERG_SYNC) and has produced succeeded snapshots for the candidate datasets (otherwise the flip is inert — the freshness gate just keeps serving PG).

Procedure

1. Inspect current flag state

npm run lakehouse:flags -- list

Shows registry defaults (all readers off) + any runtime overrides.

2. Pick candidate datasets and verify parity

Run the canary with the value-level spot-check. Target specific datasets, or let it sample synced datasets:

# specific datasets
npm run lakehouse:canary -- --smoke --dataset <uuid> --dataset <uuid>
# or a broad sample (default 20 synced datasets)
npm run lakehouse:canary -- --smoke --limit 50

Verdict gate (per dataset):

• ok — counts match and (if small) the value multiset matches → safe to flip.
• lagging — Iceberg behind but not caught up; benign (~1 min). Re-run later.
• no-snapshot — never synced; flip is inert (serves PG). Ensure the mirror runs.
• MISMATCH — a caught-up dataset disagrees → STOP, do not flip; investigate.
• wedged (summary) — a sync stuck non-succeeded → investigate before relying on it.

The script exits non-zero if any caught-up dataset mismatches.

3. Flip per-dataset first (canary scope)

npm run lakehouse:flags -- set export.stream iceberg --dataset <uuid> --note "fase1 canary"

Exercise the reader (run an export / rebuild the graph / regenerate a digest) and confirm the read-router logs reason=iceberg@eventual (and no PG-fallback warnings) for that dataset.

4. Expand scope: workspace → global

npm run lakehouse:flags -- set export.stream iceberg --workspace <uuid> --note "fase1 ws rollout"
npm run lakehouse:flags -- set export.stream iceberg --note "fase1 global"   # no scope = global

Most-specific scope wins (dataset > workspace > global), so a global flip still respects any narrower override. Roll out one reader at a time; soak between steps.

5. (Optional) promote to iceberg_only

Only for the exact full-read readers (export.stream, graph.kuzu) once parity has soaked, if you want to drop the PG fallback:

npm run lakehouse:flags -- set graph.kuzu iceberg_only --note "fase1 no-fallback"

Keep the samplers (studio.digest, studio.training_pairs) on iceberg — the fallback costs nothing and covers transient ml-runner errors.

Monitoring

• Read-router logs: [read-router] <reader> ds=<id> iceberg read failed → PG fallback: … flags real Iceberg read problems.
• Parity table: lakehouse_read_parity (written by the canary / the scheduled lib/workers/lakehouse/canary-scheduler.ts).
• Re-run the canary periodically over the flipped datasets.

Rollback (instant, ~10s)

npm run lakehouse:flags -- set export.stream off --dataset <uuid>   # force PG for one dataset
npm run lakehouse:flags -- clear export.stream                      # drop override → registry default (off)

No deploy needed; the flag cache refreshes within 10s.

Per-reader notes

• export.stream — Iceberg scan order is unspecified; a snapshot export's row ORDER may differ from PG. Acceptable for set-equivalent exports; if a consumer depends on export order, hold this reader until Phase 4 row_index. The delta/append-without-truncate export mode is already gated off for iceberg-native sources in run-export.ts.
• graph.kuzu — verify the hydrated graph node/edge counts after flipping a workspace; bulk load is order-independent.
• studio.digest / studio.training_pairs — approximate samplers; dataset-level canary parity is the correct and sufficient gate. Expect the sampled rows to differ from PG run-to-run (by design); the resulting profiles/pairs are statistically equivalent.