Summary

Appetite: 1d

Situation

Have some current code/support in datahub-next that is based on stuff joao coded up and ola improved.
- ❓ How does the current system work?
Have a Frictionless views spec
- Roughly has 2 features
  - Own ultra-simple, yaml definable mini graphing spec called simple
  - Way to connect frictionless resources with existing top-notch graphing systems like vega-lite
    - What does this mean? Well those graphing systems have their own way to specify data source and data types. In frictionless datasets or resource we already have that and we want to reuse that info.
      - Is it worth it, you can just have people respec it …
And associated render system in datahub v2
- https://datahub.io/docs/features/views ⭐ this is a pretty good intro to views in general
- The background developer info (broken link at bottom of page) should be to https://datahub.io/docs/dms/views/design ⭐⭐ (pretty useful)
Other stuff/docs. This is FYI and NOT worth reading (just needs to consolidate):
- https://datahub.io/docs/dms/views - limited conceptual overview with not much relevant for us.
- https://github.com/frictionlessdata/datapackage-render-js - the simple conversion library
- https://github.com/datopian/datapackage-views-js - not sure of status of this. have asked team.
- What's the relationship of datapackage-views-js and datapackage-render-js 🔑 See https://github.com/datopian/datapackage-views-js/issues/38 (2019):

datapackage-render-js has some utility functions for preparing the datapackage for datapackage-views-js, and datapackage-views-js has the actual way of rendering the view with react. In my experience the fact that they are 2 different libraries rather than a single one, was not very convenient - changes require two separate pull requests. Maybe there is something else to why they are separate, that I didn't notice? @anuveyatsu

Questions

What would be the simplest thing to do?
- What are the options? 🔑
  - write specs as vega lite including data link. i.e. get rid of views spec
  - have just a really basic simple spec and convert
  - get rid of the view option altogether and require users to create in their README if they want it
    - though you still have a similar need to provide ways to spec stuff in the README (e.g. is it a markdown yaml backticks block or a react component or …)
Do we want to support the simple spec (vs have people code something themselves in README)? Yes
- And only that for now (?). Yes!
- OR: maybe other specs
What library are we going to render in?
- What are the options? vega(lite) / plotly / echarts / mosaic / observale
- Will we support use of that library elsewhere and if so what are we supporting e.g. are we going to support vega lite stuff
- What are the criteria
  - Quality of line chart (or bar chart)
  - Popularity / use - see e.g. https://npmtrends.com/chart.js-vs-plotly.js-vs-vega-vs-vega-lite (chartjs higher than plotly higher than vega)
  - Ease of integration
What spec do we normalize on (if there are various simple views)? 🔑 don't normalize, correct in the source
Can we just use vega-lite?

Solution

Appendices

…

Old code to do conversion: https://github.com/frictionlessdata/datapackage-render-js

Appendix: what is the Frictionless view spec

Official spec
What we actually have:
Typescript efforts so far

The spec

https://specs.frictionlessdata.io/views/#specification

{
  // generic metadata - very similar to Data Resource or Data Package
  "name": "..."   // unique identifier for view within list of views. (should we call it id ?)
  "title": "My view" // title for this graph
  ...

  // data sources for this spec
  "resources": [ resource1, resource2 ]

  "specType": "" // one of simple | plotly | vega | vega-lite
  // graph spec
  "spec":
}

What we have

List of all datapackage.json with views: https://github.com/search?type=code&q=org%3Adatasets+specType+path%3A**%2Fdatapackage.json&p=2

Oil prices - https://github.com/datasets/oil-prices/blob/main/datapackage.json#L364C1-L377C6

"views": [
  {
    "name": "graph",
    "resourceName": "brent-daily",
    "spec": {
      "group": "Date",
      "series": [
        "Price"
      ],
      "type": "line"
    },
    "specType": "simple",
    "title": "Europe Brent Spot Price FOB (Dollars per Barrel)"
  }

Typescript

Typescript versions of the spec

// WHERE 
// original spec
export interface View {
  id?: string;
  label?: string;
  resourceName: string;
  type: string;
  state: {
    group: string;
    series: string[];
    graphType: GraphType;
  }
}

// some datapackage.json files I've found
export interface SimpleView {
  name: string;
  title: string;
  resources: string[]; // names of resources in the data package
  specType: "simple"
  spec: ViewSpec;
}

interface ViewSpec {
  type: GraphType;
  group: string;
  series: string[];
}

type GraphType = "lines-and-points" | "points" | "lines" | "bars" | "columns";

Appendix: how the current system works

https://github.com/datopian/datahub-next/blob/v2-cloud/components/frictionless-view.tsx

Converts simple spec to vega with in two parts:
- Main function does overall conversion including data link
- Conversion of graph spec itself in convertSimpleToVegaLite https://github.com/datopian/datahub-next/blob/v2-cloud/components/frictionless-view.tsx#L74-L108
Does not support specs other than simple