Dataset: epoch-data-on-ai-models

This is a Frictionless Data Package.

Concepts

Data hierarchy (from broad to specific):

• Catalog = a collection of datasets (maps to a DataHub publication, one GitHub repo)
• Dataset = a coherent data concept with a defined schema and coverage — this directory
• Data file = a concrete file artifact (csv, json, parquet…) listed as a resource in datapackage.json

Dataset lifecycle — a dataset doesn't need to be complete on day one:

• capture — just a URL or note, intent to explore
• stub — minimal entry: title, description, source link, no files yet
• archived — raw files downloaded locally
• structured — cleaned, normalised, schema documented
• enriched — analysis, visualisations, derived data added
• monitored — living source, versioned and updated over time

Catalog-as-repo pattern: if the source is a portal or collection containing many datasets (e.g. a data.gov agency, an institutional archive), give it its own repo and DataHub publication — not a subfolder here.

Structure

epoch-data-on-ai-models/
  datapackage.json   # dataset metadata and resource list
  data/              # data files (csv, json, parquet, etc.)
  .datahubignore     # files to exclude when pushing (gitignore syntax)

datapackage.json

Keep resources in sync with what's in data/:

{
  "name": "epoch-data-on-ai-models",
  "title": "Human readable title",
  "description": "What this dataset is about",
  "resources": [
    {
      "path": "data/my-file.csv",
      "name": "my-file",
      "mediatype": "text/csv"
    }
  ]
}

Workflow

# Add data files to data/
# Edit datapackage.json — update resources to list them
data pack .   # validate
dh push .     # publish to DataHub

Key rules

• Every file in data/ that you want published must be listed in resources
• name in datapackage.json must be URL-safe (lowercase, hyphens)
• Use .datahubignore to exclude scratch files, large intermediaries, etc.
• It is fine to push a stub — set lifecycle stage in datapackage.json as "status": "stub" if incomplete