Multiple Datasets in an Object Type

Overview

An object type can be backed by multiple datasets simultaneously. Each dataset contributes rows that become objects and columns that become properties. All datasets are treated equally When a new dataset is linked, its rows are synchronized into the object type:

• If a row's primary key already exists as an object, the object is enriched with data from the new dataset.
• If the primary key is new, a new object is created.

Prerequisites

Before linking a second (or subsequent) dataset to an object type, the following must be configured:

title_column is not required. If a dataset does not have the title column, objects are created using the primary key value as their name.

Linking Flow

User clicks "Add dataset"
        │
        ▼
  FileBrowserModal opens
  (already-linked datasets appear disabled)
        │
        ▼
  User selects a dataset
        │
        ▼
  MapPrimaryKeyModal opens
  (user must select which column maps to the PK)
        │
        ├── User confirms with a column selected
        │         │
        │         ▼
        │   Dataset is linked with key_column mapping
        │   Sync runs immediately
        │
        └── User closes/cancels
                  │
                  ▼
            Operation cancelled — dataset is NOT linked

Duplicate Prevention

A dataset that is already linked to the object type cannot be linked again. This is enforced at three levels:

1. Visual: the dataset appears disabled in FileBrowserModal with the label "Already linked"
2. Selection: selectItem rejects items whose id, source_id, or metadata datasetId matches an already-linked dataset
3. Guard: handleBrowseSelect verifies against the current datasets array before proceeding

Sync Engine

The sync engine (app/api/model/objects/sync.ts) runs immediately after a dataset is linked. It processes up to 10,000 rows in batches of 200.

Algorithm

1. Load all existing objects for this object type into a map:
       primary_key → { id, name, source_dataset_ids, property_overrides }

2. For each row in the new dataset:
   a. Read the value of dsKeyCol (the mapped key column)
   b. Skip if null or empty
   c. Truncate to 200 characters

   d. If primary_key EXISTS in the map:
      → ENRICH the existing object:
        - Merge row columns into property_overrides
        - Add datasetId to source_dataset_ids (if not already present)
        - Update name from title_column only if current name equals the PK

   e. If primary_key does NOT EXIST:
      → CREATE a new object:
        - name = title_column value if available, otherwise PK value
        - source_dataset_ids = [datasetId]
        - Add to in-memory map to prevent duplicate creation within same batch

3. Batch insert new objects (upsert with onConflict: object_type_id, user_id, primary_key)
4. Batch update existing objects (parallel, 50 at a time)
5. Update object_count on the object type
6. Notify Neo4j sync if any objects were created or updated

Key Column Resolution

Column Mapping (Properties)

After linking, the dataset's columns appear in the Column Mapping section. Columns are not automatically mapped to properties — the user must explicitly map them (individually or via "Automap all" in the context menu).

Property Exclusivity Rule

Each property (identified by display_name) can only be backed by one dataset. The primary key column is the only exception.

When a conflict exists:

• The "Map as property" button is replaced by an italic message: Property "X" is already backed by [dataset name]
• The toggle switch is dimmed and disabled
• "Automap all" silently skips conflicting columns

Mapping creates properties, not objects

Mapping a column to a property defines the schema — it tells the system which columns to expose as object properties. The actual object creation/enrichment happens during sync (at link time), not during column mapping.

Dataset Context Menu

Each dataset in the Column Mapping section has a three-dot menu (rendered as a portal to avoid clipping) with two options:

Data Model

object_type_datasets table

Unique constraint: (object_type_id, dataset_id) — prevents duplicate links.

objects table (relevant fields)

Unique constraint: (object_type_id, user_id, primary_key) — ensures one object per PK per user per type.