M4 Serving Deploy Runbook

Operational checklist for deploying the ML / Jupyter paradigm (PR #13) to production. Covers the order of operations, smoke validation, first-24h monitoring, and rollback. Pair with branches-and-prs.md for the notebook branch/PR feature surface.

Feature-flag status at merge time

Notebook branches + pull-requests UI ships live and visible (Phase 1 post-merge cleanup removed the NEXT_PUBLIC_NOTEBOOK_BRANCHES_ENABLED gate after the Foundry-style table redesign honestly scoped what the UI shows). Backend RPCs (merge_pull_request, archive_branch, compute_pr_diff) are atomic with FOR UPDATE locks; reading is gated by RLS on workspace_branches / pull_requests. Known gaps — rebase/conflict UI, PR reopen, cell-level diff — are tracked as Phase 3-4 of the post-M4 approach.

Model SDK + ML registry + alias serving ship live. The side-panel wizard generates Python that matches the SDK surface (Phase 2 closed the M4 drift — see _ml/snippets/snippets.drift.test.ts for the shape-level regression guard).

0. Pre-merge readiness

Confirm before clicking Merge on PR #13:

• Vercel preview build green
• pytest services/ml-runner/tests/test_invalidator.py green
• pytest services/ml-runner/tests/test_alias_resolver.py green
• pytest services/ml-runner/tests/test_predict_alias_endpoint.py green
• Staging migration applied + triggers verified (§1)
• Staging UI smoke (§2) pass

1. Deploy order

The SQL migration changes the pg_notify payload format. The Python ml-runner pods parse that payload. Deploy order matters:

1. Apply 20260703_ml_invalidation_notify.sql        ←  ml-runner old code is still tolerant
                                                        (logs "malformed payload" and skips —
                                                         worst case: missed cache invalidations
                                                         until step 2 lands)
2. Roll out ml-runner pods with new invalidator.py  ←  parser handles new payload
3. Roll out Next.js / Vercel deploy                 ←  UI fixes + workspaceId threading

During the deploy gap (step 1 → step 2), the cache invalidator will log malformed ml_alias_changed payload warnings on every notify. This is expected and self-heals as soon as step 2 completes. Cache TTL covers the missed-invalidation window.

If the gap exceeds ~30 minutes, consider step 2 as urgent — the alias cache will diverge from the registry until the parser catches up.

2. Smoke validation (staging + post-prod)

2a. Database

-- Confirm triggers exist
SELECT trigger_name, event_manipulation, event_object_table
FROM   information_schema.triggers
WHERE  trigger_name IN ('ml_alias_notify_trg', 'ml_version_archived_notify_trg');
-- Expect 2 rows.

-- Confirm GRANTs landed
SELECT grantee, privilege_type, table_name
FROM   information_schema.role_table_grants
WHERE  grantee = 'ml_runner_reader'
  AND  table_name IN ('ml_registered_models', 'ml_model_versions', 'ml_model_aliases');
-- Expect 3 SELECT grants.

2b. Live NOTIFY payload (optional but recommended)

-- Terminal A
LISTEN ml_alias_changed;

-- Terminal B — touch an existing alias row
UPDATE ml_model_aliases SET updated_at = NOW()
WHERE  alias = '<staging_alias>' LIMIT 1;

-- Terminal A should display:
-- "ml_alias_changed" with payload "<workspace_uuid>:<model_uuid>:<alias>"
-- (3 colon-separated parts — if you see 2 parts, the migration didn't apply)

2c. UI

Open the Vercel preview / staging URL → Jupyter tab → open any notebook:

• Branch selector renders Lucide <Lock> icon on protected branches (not the 🔒 emoji).
• ML side panel → tab "ML" → if any run is running, its blue status dot animates (pulse, ~1.4s period).
• Click "+ Train new model" → step 1 → type a 2+ char name → DevTools Network panel shows GET /api/ml/models?name=… firing. The React Query cache key in the request should include the real workspace UUID, not the string default.

2d. End-to-end predict path

# From the repo root with .env.local set:
npm run ml:smoke-predict     # Legacy /predict path

# M4 alias-based path (manual):
curl -X POST "$PREVIEW_URL/api/ml/predict/<model_name>@<alias>" \
  -H "Authorization: Bearer $SDK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"instances": [<...>]}'
# Expect 200 with predictions[].

3. First-24h monitoring

Prometheus metrics to graph

Logs to tail

# Confirm workspace_id is appearing in structured extras (not just in raw payload)
gcloud logging read 'jsonPayload.message="alias invalidated via notify"' \
  --limit 20 --format json | jq '.[] | .jsonPayload | {workspace_id, model_id, alias}'

# Watch for malformed payload warnings (should be zero post-deploy gap)
gcloud logging read 'jsonPayload.message="malformed ml_alias_changed payload" OR
                     jsonPayload.message="malformed ml_version_archived payload"' \
  --limit 50

The workspace_id field is the canary that the new pipeline is end-to-end: SQL trigger → NOTIFY → Python parser → log extras.

4. Known failure modes

4a. malformed ml_alias_changed payload warnings keep firing

Cause: the SQL migration was rolled back or never applied, but the ml-runner is on the new code path. Old payload model_id:alias (2 parts) arrives where the parser expects 3.

Fix: re-apply 20260703_ml_invalidation_notify.sql. The parser ignores the bad notifies until the migration takes effect; no data corruption.

4b. Cache invalidations don't fire after an alias change

Cause: the ml-runner's LISTEN connection dropped and didn't reconnect.

Symptom: the metric alias_invalidations_total stops incrementing even when ml_model_aliases is being edited.

Fix:

# Restart the ml-runner pod(s):
kubectl rollout restart deployment/ml-runner -n <ns>
# Or check the invalidator logs for "cache invalidator connection lost".
# The exponential backoff caps at 30s — if reconnect is failing past that
# window, suspect Postgres-side connection limits or network ACL.

4c. UI ML wizard step 1 shows Checking… forever

Cause: workspaceId is empty (Clerk metadata not yet hydrated).

Symptom: the React Query for GET /api/ml/models?name=… never fires (query is enabled: false). Step 1 "next" button works but availability hint stays in the typing state.

Fix: confirm Clerk session injects publicMetadata.workspaceId. This is platform-level, not an ml-runner concern.

4d. Signed URL minting fails on cold-cache predict

Symptom: 502 ARTIFACT_DOWNLOAD_FAILED on /predict/alias for models not in cache.

Cause: SUPABASE_SERVICE_KEY or SUPABASE_URL not set in the ml-runner env; or the storage bucket doesn't have the artifact.

Fix: verify env vars and the artifact path in ml_model_versions.artifact_uri.

5. Rollback

If post-deploy issues are severe, rollback in reverse order:

# 1. Revert Next.js / Vercel — re-deploy the previous commit
#    (UI fixes are forward-compatible with the old parser, so this
#     step is optional; revert only if a UI regression is the cause)

# 2. Roll the ml-runner back to the previous image tag
kubectl set image deployment/ml-runner \
  ml-runner=<registry>/ml-runner:<previous_tag> -n <ns>

# 3. Drop the triggers & functions added by 20260703
psql $DATABASE_URL <<'SQL'
DROP TRIGGER IF EXISTS ml_alias_notify_trg            ON public.ml_model_aliases;
DROP TRIGGER IF EXISTS ml_version_archived_notify_trg ON public.ml_model_versions;
DROP FUNCTION IF EXISTS public.ml_alias_notify_fn();
DROP FUNCTION IF EXISTS public.ml_version_archived_notify_fn();
REVOKE SELECT ON public.ml_registered_models FROM ml_runner_reader;
REVOKE SELECT ON public.ml_model_versions    FROM ml_runner_reader;
REVOKE SELECT ON public.ml_model_aliases     FROM ml_runner_reader;
NOTIFY pgrst, 'reload schema';
SQL

The migration is idempotent on re-apply (DROP IF EXISTS + CREATE OR REPLACE), so re-deploying after a rollback is safe.

6. Cleanup after stability is confirmed

After ~72h of clean metrics post-merge:

# Delete absorbed branches (already closed on remote, no PRs open)
git push origin --delete feature/ml-milestone-1
git push origin --delete feature/ml-milestone-3
git push origin --delete Building-Model-SDK
git push origin --delete feature/jupyter-branches-ui
git push origin --delete "feature/(ML)Python-SDK-node.ml"

Or leave them in place as historical reference — they cost nothing on the remote.