`node.transforms` SDK — v1 contract

Status: Phases 1–5 shipped + Hardening P0–P3 + P5–P8 done. SDK is pre-installed in the Hub image (services/jupyterhub/Dockerfile pip-installs from services/jupyterhub/node-sdk/ during the build). Current SDK version: 0.5.0.

Owners: platform/jupyter integration.

0. Where the truth lives

The wire contract is defined in docs/sdk-v1.openapi.yaml.

That OpenAPI 3.1 document is the source of truth for every endpoint, schema, header, and error code the SDK speaks. This narrative doc explains the why behind the design (architecture, hardening phases, design decisions); the YAML is what TS handlers and the Python kernel client must agree with on the wire.

Drift between the YAML, lib/sdk/types.ts, and services/jupyterhub/node-sdk/node/transforms.py is caught automatically by:

npx tsx scripts/check-sdk-spec.ts
# or
npm run check:sdk-spec

The validator covers four drift modes:

Error codes — SdkErrorCode enum in spec ↔ SDK_ERROR_STATUS map in TS.
TS headers — every SDK_HEADER_* constant in TS appears in the spec.
Python headers — every _HEADER_* constant in the kernel client appears in the spec.
Routes — every app/api/sdk/**/route.ts on disk has a matching spec path, and vice versa.

Run the validator before any wire-level change lands. Adding a new endpoint or error code without updating the YAML fails CI.

1. What is this

A Python SDK installed in every notebook kernel that lets user code move data between the Node workspace and the kernel's memory:

from node.transforms import Dataset

# bring data IN
animals = Dataset.get("animals").read_table(format="pandas")

# push results OUT
Dataset.get("animals_clean").write_table(animals.dropna())

The SDK is the programmatic surface. UI endpoints (/api/datasets/..., /api/notebooks/...) keep evolving with product iterations; the SDK exposes a stable, versioned API (/api/sdk/v1/...) that user-written notebooks can rely on without breaking under our refactors.

End-to-end role:

┌──────────────────────────┐
│ Workspace data           │
│ (datasets, dataset_rows) │
└──────────┬───────────────┘
           │ read_table()
           ▼
┌──────────────────────────┐
│ Notebook kernel          │
│   df = ...               │
│   df_clean = transform() │
└──────────┬───────────────┘
           │ write_table()
           ▼
┌──────────────────────────┐
│ Workspace data (new txn) │
│ → consumed by:           │
│   • object types         │
│   • dashboards           │
│   • action types         │
│   • other notebooks      │
└──────────────────────────┘

2. Decisions (already made — do not relitigate without ADR update)

Decision	Choice	Why not the alternatives
Naming	`node.transforms`	Foundry parity — DS coming from there feels at home; `node.*` is our namespace.
Auth	Per-named-server JWT signed by the platform	Clerk JWTs expire too fast for an interactive session, and exposing the user JWT to user-written code is an exfiltration vector. Workspace-shared API key has no audit trail of who/where.
HTTP namespace	`/api/sdk/v1/...` (versioned, separate from UI)	UI routes evolve at product cadence; SDK contract has to be stable. Versioning lets us ship `/v2/` without breaking notebooks.
Write semantics	Transactional replace (atomic)	`dataset_rows.transaction_id` already exists in the schema. Replace covers ~90% of use; append/incremental land later when a real use case appears.
Distribution	Pre-installed in the Hub Docker image	"Professional, traceable" workspace = users don't `pip install` our SDK every kernel. Same model as Foundry.

3. Auth flow

Hub spawn (pre_spawn_hook in jupyterhub_config.py):
  1. Reads auth_state populated by ClerkAuthenticator:
     { workspace_id, user_id, role, clerk_token }
  2. notebook_id = named server name (= Node artifact UUID)
  3. POST {NODE_API_BASE}/api/sdk/internal/mint-kernel-token
       Authorization: token <JUPYTERHUB_PLATFORM_TOKEN>
       Body: { workspaceId, notebookId, userId, ttlSec: 86400 }
     → 200 { token, expiresAt }
  4. Inject env vars on the kernel process:
       NODE_API_BASE       (e.g. https://app.paladio.io)
       NODE_API_TOKEN      (the JWT)
       NODE_TOKEN_EXPIRES_AT  (ISO timestamp, optional, hint for SDK)

Kernel runtime (SDK):
  5. node.transforms imports os, reads NODE_API_BASE + NODE_API_TOKEN.
  6. Every API call carries `Authorization: Bearer <NODE_API_TOKEN>`.

Server-side (Next.js middleware on /api/sdk/v1/*):
  7. Verify JWT signature with NODE_SDK_SIGNING_KEY (HS256).
  8. Extract claims { sub, ws, nb, iat, exp }.
  9. Reject if exp < now (401).
 10. Reject if claim ws ≠ resolved dataset's workspace (403).
 11. Pass claims into the handler's typed context.

JWT claims shape (HS256)

{
  "sub":  "<clerk_user_id>",
  "ws":   "<workspace_uuid>",
  "nb":   "<notebook_uuid>",
  "iat":  1700000000,
  "exp":  1700086400,
  "iss":  "node-platform",
  "aud":  "node-sdk"
}

TTL: 24 h. After expiry the user pauses + resumes the named server (which re-mints). Tying the auth lifetime to the named server lifecycle means there is no in-kernel refresh dance.

Required env vars

On the Hub deployment (Railway):

NODE_API_BASE — public Node URL (https://app.paladio.io).
JUPYTERHUB_PLATFORM_TOKEN — already exists, reused for the mint call.

On the Next.js deployment (Vercel):

NODE_SDK_SIGNING_KEY — symmetric secret for HS256 signing. Long random string, rotated by re-deploying both sides.
JUPYTERHUB_PLATFORM_TOKEN — used by the mint endpoint to verify the caller is the Hub.

4. HTTP API contract

All endpoints under /api/sdk/v1/* require Authorization: Bearer <kernel JWT> unless noted otherwise. Errors follow { error: "<CODE>", message: "<human>" }.

4.1 Internal — token issuance

POST /api/sdk/internal/mint-kernel-token
Auth: Authorization: token <JUPYTERHUB_PLATFORM_TOKEN>
Body: {
  workspaceId: string  (uuid),
  notebookId:  string  (uuid),
  userId:      string  (clerk user_id),
  ttlSec?:     number  (default 86400, max 86400)
}
Response 200: {
  token:       string   (HS256 JWT),
  expiresAt:   string   (ISO 8601)
}
Errors:
  401 PLATFORM_TOKEN_INVALID
  400 INVALID_INPUT

4.2 Identity probe

GET /api/sdk/v1/whoami
Response 200: {
  workspaceId: string,
  notebookId:  string,
  userId:      string,
  expiresAt:   string
}

Used by the SDK to verify connectivity + by tests; not part of the user-facing API surface.

4.3 Dataset schema

GET /api/sdk/v1/datasets/{nameOrId}/schema
Response 200: {
  id:           string,
  name:         string,
  workspaceId:  string,
  columns: [
    { name: string, type: string, nullable: boolean }
  ],
  rowCount:     number,
  currentTransactionId: string | null
}
Errors:
  404 DATASET_NOT_FOUND
  403 WORKSPACE_MISMATCH       (token ws ≠ dataset ws)

type is the column type as stored in datasets.schema jsonb (text, number, boolean, timestamp, json, etc.). A canonical type vocabulary lives in §6.

4.4 Read rows

GET /api/sdk/v1/datasets/{nameOrId}/rows?limit=<N>&offset=<M>
Response 200: {
  id:           string,
  name:         string,
  schema:       Column[],
  rows:         Record<string, any>[],
  rowCount:     number,
  hasMore:      boolean,
  transactionId: string,
  pagination:   { limit, offset }
}
Defaults: limit=10000, offset=0. Hard cap: limit ≤ 100000.

Future: Apache Arrow IPC binary response when Accept: application/vnd.apache.arrow.stream. v1 ships JSON only.

4.5 Write — transactional replace

Three endpoints make up a write:

POST /api/sdk/v1/datasets/{nameOrId}/transactions
Body: { mode: "replace" }    // "append" reserved
Response 200: {
  transactionId: string,
  createdAt:     string,
  mode:          "replace"
}

POST /api/sdk/v1/datasets/{nameOrId}/transactions/{txnId}/rows
Body: { rows: Record<string, any>[] }    // batched; 10k per call recommended
Response 200: {
  transactionId: string,
  rowsWritten:   number,
  totalSoFar:    number
}

POST /api/sdk/v1/datasets/{nameOrId}/transactions/{txnId}/commit
Response 200: {
  transactionId:           string,
  committedAt:             string,
  totalRows:               number,
  previousTransactionId:   string | null
}

And the abort path (called by the SDK on exception):

POST /api/sdk/v1/datasets/{nameOrId}/transactions/{txnId}/abort
Response 200: { aborted: true, rowsDiscarded: number }

Errors across the trio:

409 TRANSACTION_CLOSED — txn already committed/aborted
409 SCHEMA_MISMATCH — rows shape diverges from datasets.schema
403 WORKSPACE_MISMATCH

5. Python SDK contract

Public surface (everything else is private):

from node.transforms import Dataset, Column, NodeError, ...

ds = Dataset.get("animals")           # → Dataset handle (no IO)
ds.id, ds.name, ds.workspace_id       # cheap properties (cached)
ds.schema()                           # → list[Column], one HTTP roundtrip
df = ds.read_table(format="pandas")   # → DataFrame, materialises rows
ds.write_table(df)                    # → str txnId, atomic replace
ds.versions()                         # → list[Transaction], reserved

with ds.transaction() as tx:          # multi-batch writes (reserved)
    tx.append(chunk_1)
    tx.append(chunk_2)
# commit on exit, abort on exception

Errors form a hierarchy:

NodeError                        # base
├── NodeAuthError                # 401 / token bad
├── NodeWorkspaceError           # 403 / ws mismatch
├── DatasetNotFoundError         # 404
├── SchemaMismatchError          # 409 SCHEMA_MISMATCH
├── TransactionConflictError     # 409 TRANSACTION_CLOSED
└── NodeNetworkError             # transport-layer failure (after retries)

Full type-hinted signatures live in services/jupyterhub/node-sdk/node/transforms.py. The SDK source is co-located with the Hub Dockerfile so the build context is hermetic — every kernel image pip-installs from this path at build time. There is no separate package registry; rebuilding the Hub image is how you roll out a new SDK version.

6. End-to-end traces

Read

Notebook code:
  df = Dataset.get("animals").read_table(format="pandas")

SDK:
  GET /api/sdk/v1/datasets/animals/rows
    Authorization: Bearer <NODE_API_TOKEN>

Next.js handler:
  • verify JWT → claims { ws, nb, sub }
  • resolve "animals" → datasets.id (RLS-bypassing service client; the
    workspace check below enforces tenancy)
  • assert dataset.workspace_id === claims.ws  (or 403)
  • SELECT * FROM dataset_rows
      WHERE dataset_id = <id>
        AND transaction_id = (SELECT current_transaction_id FROM datasets WHERE id = <id>)
      ORDER BY row_index
      LIMIT 10000
  • return { schema, rows, transactionId }

SDK:
  • pd.DataFrame.from_records(rows)
  • returns to caller

Write

Notebook code:
  Dataset.get("animals_clean").write_table(df)

SDK:
  POST /api/sdk/v1/datasets/animals_clean/transactions { mode: "replace" }
    → { transactionId: T1 }

  for chunk in batches_of(df, 10000):
    POST /api/sdk/v1/datasets/animals_clean/transactions/T1/rows { rows: chunk }

  POST /api/sdk/v1/datasets/animals_clean/transactions/T1/commit
    → { committedAt, totalRows, previousTransactionId }

Next.js handler at commit:
  BEGIN;
  UPDATE datasets
     SET current_transaction_id = T1, updated_at = now()
   WHERE id = <id>
     AND workspace_id = <claims.ws>;
  COMMIT;

  Old rows (transaction_id != T1) remain in dataset_rows as history.
  Reads filter by current_transaction_id, so only T1's rows are visible.

  If the notebook crashes mid-write, the txn is left in dataset_rows
  but never becomes current. A periodic cleanup worker (future phase)
  reaps orphaned transactions older than N days.

7. Schema additions needed

One migration in Phase 3:

-- 20260620_datasets_current_transaction.sql
ALTER TABLE public.datasets
  ADD COLUMN IF NOT EXISTS current_transaction_id uuid;

CREATE INDEX IF NOT EXISTS idx_datasets_current_transaction
  ON public.datasets(current_transaction_id);

-- Backfill: pick the most-recently-written transaction_id per dataset.
UPDATE public.datasets d
   SET current_transaction_id = (
     SELECT transaction_id
       FROM public.dataset_rows r
      WHERE r.dataset_id = d.id
      ORDER BY r.created_at DESC
      LIMIT 1
   )
 WHERE current_transaction_id IS NULL;

The UI's existing reads of dataset_rows need to filter by current_transaction_id after this lands. That's a one-shot ripple; catalogued in Phase 3.

8. Canonical column type vocabulary (v1)

Wire type	Pandas dtype	Polars dtype	Notes
`text`	`object` (string)	`Utf8`	Default for unknown / mixed
`number`	`float64`	`Float64`	All numeric (int + float collapse)
`integer`	`Int64`	`Int64`	Optional refinement of `number`
`boolean`	`bool`	`Boolean`
`timestamp`	`datetime64[ns, UTC]`	`Datetime`	ISO 8601 strings on the wire
`json`	`object`	`Object`	Nested structures

The wire shape always carries values as JSON-serializable scalars (strings, numbers, bools, null) plus optional nested arrays/objects for json columns. Conversion to pandas/polars happens SDK-side.

9. Open questions / deferred

Concurrent writes: two notebooks committing transactions to the same dataset — last-write-wins. v1 accepts this; v2 may add optimistic concurrency (If-Match: <currentTxnId> header).
Apache Arrow wire format: nice-to-have for big datasets; reserved for a v1.x patch when read latency on >1M rows hurts.
Lineage capture: every write should record read_table() upstream calls + commit hash of the notebook → builds the DAG. Owns its own ADR; deferred to a later phase.
Read-only roles: v1 SDK trusts the JWT's claims. If the user is a viewer they should get 403 on writes — enforce in the middleware.
Append / incremental writes: the mode: "append" field on POST /transactions is reserved but rejects with 400 in v1.

10. Phase plan recap

Phase 0 (this doc) — contract, no code. ✅
Phase 1 — auth foundation: token mint endpoint, JWT verify middleware, whoami probe, spawner injects env vars. ✅
Phase 2 — read endpoints + SDK read-side. ✅
Phase 3 — write endpoints + SDK write-side + migration 20260620_sdk_transactions.sql. ✅
Phase 4 — polish: lastCommittedAt on /schema, new /versions endpoint, structured error attributes (.code/.status_code), GET retries with exponential backoff, schema-aware dtype coercion in read_table, Dataset.row_count/current_transaction_id/last_committed_at properties. ✅
Phase 5 — distribution: SDK source moved to services/jupyterhub/node-sdk/ and pip-installed during the Hub Docker build with a from node.transforms import Dataset smoke test. Rebuild the image to roll out a new SDK version. ✅

Each phase shipped independently. Read-only state after Phase 2 was already useful as exploration tooling; Phase 3 closed the workspace ↔ kernel ↔ workspace cycle; Phase 4 made the SDK pleasant to use; Phase 5 means notebook authors can from node.transforms import Dataset without ever running pip install.

node.transforms SDK — v1 contract