Data Explorer

The Datopian Data Explorer is a React single page application and framework for creating and displaying rich data explorers (think Tableau-lite). Use stand-alone or with CKAN. For CKAN it is a drop-in replacement for ReclineJS in CKAN Classic.

Data Explorer

Data Explorer for the City of Montreal

Features / Highlights

"Data Explorer" is an embedable React/Redux application that allows users to:

Explore tabular, map, PDF, and other types of data
Create map views of tabular data using the Map Builder
Create charts and graphs of tabular data using Chart Builder
Easily build SQL queries for Data Store API using graphical interface of Datastore Query Builder

Components

The Data Explorer application acts as a coordinating layer and state management solution – via Redux – for several libraries, also maintained by Datopian.

Datapackage Views

Datapackage View is the rendering engine for the main window of the Data Explorer.

The above image displays a table shown at the Table tab, but note that Datapackage-views renders all data visualizations: Tables, Charts, Maps, and others.

Datastore Query Builder

The Datastore Query Builder interfaces with the Datastore API to allow users to search data resources using an SQL like interface. See the docs for this module here - Datastore Query Builder docs.

Map Builder

Map Builder allows users to build maps based on geo-data contained in tabular resources.

Supported geo formats:

lon / lat (separate columns)

Chart Builder

Chart Builder allows users to create charts and graphs from tabular data.

Quick-start (Sandbox)

Clone the data explorer

$ git clone git@gitlab.com:datopian/data-explorer.git

Use yarn to install the project dependencies

$ cd data-explorer
$ yarn

To see the Data Explorer running in a sandbox environment run Cosmos

$ yarn cosmos

Configuration

data-datapackage attribute may influence how the element will be displayed. It can be created from a datapackage descriptor.

Fixtures

Until we have better documentation on Data Explorer settings, use the Cosmos fixtures as an example of how to instantiate / configure the Data Explorer.

Serialized state

store->serializedState is a representation of the application state without fetched data A data-explorer can be "hydrated" using the serialized state, it will refetch the data, and will render in the same state it was exported in

Share links can be added in datapakage.resources[0].api attribute.

There is common limit of up 2000 characters on URL strings. Our share links contain the entire application store tree, which is often larger than 2000 characters, in which the application state cannot be shared via URL. Thems the breaks.

Translations

Add a Translation To Data Explorer

To add a translation to a new language to the data explorer you need to:

clone the repository you need to update

git clone git@gitlab.com:datopian/data-explorer.git

go to src/i18n/locales/ folder
add a new sub-folder with locale name and the new language json file (e.g. src/i18n/locales/ru/translation.json)
add the new file to resources settings in i18n.js: src/i18n/i18n.js:

import en from './locales/en/translation.json'
import da from './locales/da/translation.json'
import ru from './locales/ru/translation.json'
    ...
      ru: {
        translation: {
          ...require('./locales/ru/translation.json'),
          ...
        }
      },
    ...

create a merge request with the changes

Add a translation To a Component

Some strings may come from a component, to add translation for them will require some extra steps, e.g. datapackage-views-js:

clone the repository

https://github.com/datopian/datapackage-views-js.git

go to src/i18n/locales/ folder
add a new sub-folder with locale name and the new language json file (e.g. src/i18n/locales/ru/translation.json)
add the new file to resources settings in i18n.js: src/i18n/i18n.js:

...
import ru from './locales/ru/translation.json'
    ...
    resources: {
      ...
      ru: {translation: ru},
    },
    ...

create a pull request for datapackage-views-js
get the new datapackage-views-js version after merging (e.g. 1.3.0)
clone data-explorer
upgrade the data-explorer's datapackage-views-js dependency with the new version: a. update package.json b. run yarn install
add the component's translations path to Data Explorer:

import en from './locales/en/translation.json'
import da from './locales/da/translation.json'
import ru from './locales/ru/translation.json'
    ...
      ru: {
        translation: {
          ...require('./locales/ru/translation.json'),
          ...require('datapackage-views-js/src/i18n/locales/ru/translation.json'),
        }
      },
    ...

create a merge request for data-explorer

Testing a Newly Added Language

To see your language changes in Data Explorer you can run yarn start and change the language cookie of the page (defaultLocale):

i18n Cookie

Language detection

Language detection rules are determined by detection option in src/i18n/i18n.js file. Please edit with care, as other projects may already depend on them.

Embedding in CKAN NG Theme

Copy bundle files to theme's `public` directory

$ cp data-explorer/build/static/js/*.js frontend-v2/themes/your_theme/public/js
$ cp data-explorer/build/static/js/*.map frontend-v2/themes/your_theme/public/js
$ cp data-explorer/build/static/css/* frontend-v2/themes/your_theme/public/css

Note on app bundles

The bundled resources have a hash in the filename, for example 2.a3e71132.chunk.js

During development it may be preferable to remove the hash from the file name to avoid having to update the script tag during iteration, for example

$ mv 2.a3e71132.chunk.js 2.chunk.js

A couple caveats:

The .map file names should remain the same so that they are loaded properly
Browser cache may need to be invalidated manually to ensure that the latest script is loaded

Require Data Explorer resources in NG theme template

In /themes/your-theme/views/your-template-wth-explplorer.html

<!-- Everything before the content block goes here -->
{% block content %}

<!-- Data Explorer CSS -->
<link rel="stylesheet" type="text/css" href="/static/css/main.chunk.css">
<link rel="stylesheet" type="text/css" href="/static/css/2.chunk.css">
<!-- End Data Explorer CSS -->

Configure datapackage

<!-- where datapackage is -->
<srcipt>
  const datapackage = {
    resources: [{resource}], // single resource for this view
    views: [...], // can be 3 views aka widgets
    controls: { 
      showChartBuilder: true,
      showMapBuilder: true 
    }
  }
</srcipt>

Add data-explorer tags to the page markup

Each Data Explorer instance needs a corresponding <div> in the DOM. For example:

{% for resource in dataset.resources %}
  <div class="data-explorer" id="data-explorer-{{ loop.index - 1 }}" data-datapackage='{{ dataset.dataExplorers[loop.index - 1] | safe}}'></div>
{% endfor %}

Note that each container div needs the following attributes:

class="data-explorer" (All explorer divs should have this class)
id="data-explorer-0" (1, 2, etc…)
data-datapackage={JSON CONFIG}` (A valid JSON configuration)

Add data explorer scripts to your template

<script type="text/javascript" src="/static/js/runtime~main.js"></script>
<script type="text/javascript" src="/static/js/2.chunk.js"></script>
<script type="text/javascript" src="/static/js/main.chunk.js"></script>

NOTE that the scripts should be loaded after the container divs are in the DOM, typically by placing the <script> tags at the bottom of the footer

See a real-world example here

New builds

In order to build files for production, run npm run build or yarn build.

You need to have node version >= 12 in order to build files. Otherwise a 'heap out of memory error' gets thrown.

Component changes

If the changes involve component updates that live in separate repositories make sure to upgrade them too before building:

Prepare the component with dist version (eg run yarn build:package in the component repo, see this for an example)
run yarn add <package> to get latest changes, e.g. yarn add @datopian/datastore-query-builder (do not use yarn upgrade, see here on why https://github.com/datopian/data-explorer/issues/28#issuecomment-700792966)
you can verify changes in yarn.lock - there should be the latest component commit id
yarn build in data-explorer

Testing not yet released component changes

If there are some changes to be tested that are not ready to be released in a component the best option is to use cosmos directly in the component repository, but if that is not enough you can add the dependency from a branch temporarily:

yarn add https://github.com/datopian/datastore-query-builder.git#<branch name>

Appendix: Design

See Data Explorer Design page »

Data Explorer

Features / Highlights

Components

Datapackage Views

Datastore Query Builder

Map Builder

Chart Builder

Quick-start (Sandbox)

Configuration

Fixtures

Serialized state

Share links

Translations

Add a Translation To Data Explorer

Add a translation To a Component

Testing a Newly Added Language

Language detection

Embedding in CKAN NG Theme

Copy bundle files to theme's public directory

Note on app bundles

Require Data Explorer resources in NG theme template

Configure datapackage

Add data-explorer tags to the page markup

Add data explorer scripts to your template

New builds

Component changes

Testing not yet released component changes

Appendix: Design

Copy bundle files to theme's `public` directory