Re notes/data-presentation|Data Presentation in DataHub incl Literate etc

Re Data Presentation in DataHub incl Literate etc

../people/rufuspollock

#todo/process

Summary

Data presentation in our apps can be organized in two major ways:

Direct load: raw data is loaded (and queried) by JS code in the frontend and then presented
Via Data API raw data is first imported to data storage with a data API around it and then the frontend presentation access that API

See below for details.

Choosing if we want to support one or both of these is a crucial implementation choice.

Key choices to make:

Do we go a "data api" only route? 🚧2023-02-15 initial KISS intuition is yes …
If we do, do we bother to "factor" out cleanly views stuff e.g. to have a TableView which takes a well defined object like a Frictionless Resource? **🚧2023-02-15 probably not that much **

Implications

If we go data API only we don't have support in e.g. flowershow (though it could be reasonably easy to add i guess at least for very simple case like csv?)

Common stuff

What table library do we use? ✅2023-02-15 react-table seems definitely best headless now. let's use it
What chart library do we use? 🚧2023-02-15 choose between vega, plotly and chartjs …?
Do we do mapping (at all)? 🚧2023-02-15 probably yes, at least a bit. it's quite easy.

thoughts

even if we do data api only will want to be able to develop against data so suspect we want a simple way to load csv with a schema??

Question tree

Two ways data can flow into data presentation

2 ways raw data can flow into data presentation:

Direct load: raw data is loaded (and queried) by JS code in the frontend
Via Data API raw data is first imported to data storage with a data api around it

Raw data is a data file (e.g. csv, xls, etc) on disk or online somewhere. Emphasize raw data b/c of course frontend renderer will ultimately have to load data from somewhere in some format.

Direct load

via Data API

The essence of the difference

The essence of the difference is whether it is frontend or backend that is responsible for:

Loader: i.e. importing (i.e. converting the raw data to a standarized, well structured form)
Query: querying over data i.e. providing some way to filter etc

Note: for large data support the Data API route is only option as the direct route simply isn't feasible (other than for a sample preview where you load only a part of the file). e.g. loading a 1GB CSV into memory in your browser won't go well. However, handling 1GB CSV in postgres is easy and so a Data API can work.

Pros and cons of Data API (and direct load)

data importing and prep can be done in specialist backend code
clean separation: frontend has clear guarantees about how to access data
large data is no problem: many highly developed backends for data ranging from MBs to PBs. frontend can interact with even very large datasets via querying.
data integration: merging or integrating datasets can be done in the backend

Cons: a data API has to be created and maintained. =>

no simple static apps (and more complexity in general)
if backend code fails the presentation fails
no live interaction with data (data has to be loaded first to API and then becomes live in frontend)

NB: from a technical/architecture perspective the issue is that a) the query components in the frontend are usually quite tied to the backend API b) the query stuff is a large part of the value add (it's what makes something and "explorer" rather than just a sample view). Whilst a simple view is nice, what everyone ends up wanting is an explorer.

To summarize:

if you can go with a data API, do it
If you are handling largish data you have you have choice other than Data API
BUT: if you want something simple that works locally and is easy to publish especially statically then you want direct …

Comments:

if i were doing direct today i would probably try and get stuff into sqlite and then run against that (rather than try and build my own query layer). there are now some pretty cool ways to run sqlite in the browser.
- you would then build the sqlite as part of the static build
- OR: i would try alasql or something
for the Data API route i'd strongly examine just wholesale adopting some cloud provider solution off the shelf e.g. google bigquery is great.
also: extracting a small sample e.g. first 500 rows and converting to json and caching that separate from data API can still support some simple direct load and provides a good fallback if data api has an issue (or just to save queries e.g. default view just uses that until a user explicitly queries)

Job Stories

Data Literate case

I have markdown + data (csv, json, xlsx, sqlite?) and want to put it online with the minimum of fuss and complexity so that others can see and use my work

I have a dataset i want to publish consisting of README plus a data file (or files) e.g. in CSV
I have a data-driven story where i want to display data and/or charts in my markdown doc
I have a data science analysis i quickly want to publish

On disk you have:

README.md
data.csv

In README i have something like:

# My Amazing Dataset

This is my awesome dataset / data driven analysis.

Here's the data

<DataTable src="mydata.csv" />

You can create a nice chart like this:

<LineChart config={} data="mydata.csv" />

And pfft in a puff of smoke this turns into a nice rendered page with a table and Line Chart. Here was the demo:

https://portaljs.org/data-literate/demo

Inbox

back post data literate idea (https://portaljs.org/data-literate) to datahub blog
- port over demo to datahub.io? this involves a load of components?
  - Aside: for this demo what i really want is "side by side" showcase with markdown on one side and the rendering result on other side ( and this be chunk by chunk in the way e.g. https://slatedocs.github.io/slate/#get-all-kittens does it)