Data Transform examples on Global CO2 Emissions from Fossil Fuels since 1751

examples

Files Size Format Created Updated License Source
1 19kB csv ODC-PDDL CDIAC GitHub repo
This is an example Data Package to demonstrate how data transforms works. In this example, we explain how filtering and applying formula can be done before data package gets rendered in showcase page. It assumes publisher is already familiar with Data Packages and views specifications (views read more
Download

Data Files

global  

Field information

Field Name Order Type (Format) Description
Year 1 date (%Y-%m-%d) Year
Total 2 number Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)
Gas Fuel 3 number Carbon emissions from gas fuel consumption
Liquid Fuel 4 number Carbon emissions from liquid fuel consumption
Solid Fuel 5 number Carbon emissions from solid fuel consumption
Cement 6 number Carbon emissions from cement production
Gas Flaring 7 number Carbon emissions from gas flaring
Per Capita 8 number Per capita carbon emissions (metric tons of carbon; after 1949 only)

Read me

This is an example Data Package to demonstrate how data transforms works. In this example, we explain how filtering and applying formula can be done before data package gets rendered in showcase page. It assumes publisher is already familiar with Data Packages and views specifications (views property in Data Package specifications).

Transforming data

Data transforms are specified in resources property in views. Each resource is an object that contains following attributes:

  • "specType": "table" - this way we define the view as a table (other options are "simple" (renders a graph and accepts Plotly spec) and "vega" (renders a graph and accepts Vega spec)).
  • "resources" property is an array of objects in this case, where publishers can define data transforms they want to apply.
  • "name" - name of the resource as a reference.
  • "transform" - array of transforms. Each transform is an object, which properties vary depending on transform type. Only common property is "type" that is used to specify transform type.

Filtering data

Under the graph on the top of this page, you can find a table that displays filtered data. Raw data is displayed in preview section. As you can see we are filtering by year - showing data for 2000 and onwards. This is described in the second view object of views property:

  • "type": "filter" - this way we define the transform to be a filter.
  • "expression" - any JavaScript expression that evaluates to Boolean using some field(s) from the resource. E.g., in our example we are using "Year" field - converting it into JS date object then getting year by using getFullYear() method and filtering by larger or equal sign.

Applying formula

Under the filtered data table, there is another table that displays raw data but with column “Gas Fuels” as a percentage of total figures. JSON representation of it can be found in the third view object:

  • "type": "formula" - defining transform type to be used.
  • "expressions" - list of any JavaScript expressions that will be applied to specified data field.
  • "asFields" - list of field names. This should be used according to list of expressions.

Descriptor for this data package

{
  "datahub": {
    "findability": "published",
    "hash": "05c235c0506fb2ce9228806ff895d3ac",
    "modified": "2017-08-08T17:37:44.897182",
    "owner": "examples",
    "ownerid": "examples",
    "stats": {
      "bytes": 19359,
      "rowcount": 0
    }
  },
  "id": "examples/transform-examples-on-co2-fossil-global",
  "license": "ODC-PDDL",
  "name": "transform-examples-on-co2-fossil-global",
  "profile": "data-package",
  "readme": "This is an example Data Package to demonstrate how data transforms works. In this example, we explain how filtering and applying formula can be done before data package gets rendered in showcase page. It assumes publisher is already familiar with Data Packages and views specifications (`views` property in Data Package specifications).\n\n### Transforming data\n\nData transforms are specified in `resources` property in `views`. Each resource is an object that contains following attributes:\n\n* `\"specType\": \"table\"` - this way we define the view as a table (other options are `\"simple\"` (renders a graph and accepts Plotly spec) and `\"vega\"` (renders a graph and accepts Vega spec)).\n* `\"resources\"` property is an array of objects in this case, where publishers can define data transforms they want to apply.\n* `\"name\"` - name of the resource as a reference.\n* `\"transform\"` - array of transforms. Each transform is an object, which properties vary depending on transform type. Only common property is `\"type\"` that is used to specify transform type.\n\n### Filtering data\n\nUnder the graph on the top of this page, you can find a table that displays filtered data. Raw data is displayed in preview section. As you can see we are filtering by year - showing data for 2000 and onwards. This is described in the second view object of `views` property:\n\n* `\"type\": \"filter\"` - this way we define the transform to be a filter.\n* `\"expression\"` - any JavaScript expression that evaluates to Boolean using some field(s) from the resource. E.g., in our example we are using `\"Year\"` field - converting it into JS date object then getting year by using `getFullYear()` method and filtering by larger or equal sign.\n\n### Applying formula\n\nUnder the filtered data table, there is another table that displays raw data but with column \"Gas Fuels\" as a percentage of total figures. JSON representation of it can be found in the third view object:\n\n* `\"type\": \"formula\"` - defining transform type to be used.\n* `\"expressions\"` - list of any JavaScript expressions that will be applied to specified data field.\n* `\"asFields\"` - list of field names. This should be used according to list of expressions.\n\n### Descriptor for this data package\n\n{{ dp.json }}\n",
  "resources": [
    {
      "bytes": 14727,
      "datahub": {
        "derivedFrom": [
          "global"
        ],
        "type": "derived/csv"
      },
      "dialect": {
        "delimiter": ",",
        "doubleQuote": true,
        "lineTerminator": "\r\n",
        "quoteChar": "\"",
        "skipInitialSpace": false
      },
      "dpp:streamedFrom": "https://s3.amazonaws.com/rawstore.datahub.io/09015539c2fc32bb3c4afead7df461b5#.csv",
      "encoding": "utf-8",
      "format": "csv",
      "hash": "a315f7352a1dd718da3677e2620998e6",
      "name": "global",
      "path": "https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global:global_csv/data/global_csv.csv",
      "profile": "data-resource",
      "rowcount": 260,
      "schema": {
        "fields": [
          {
            "description": "Year",
            "format": "%Y-%m-%d",
            "name": "Year",
            "type": "date"
          },
          {
            "decimalChar": ".",
            "description": "Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)",
            "groupChar": "",
            "name": "Total",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Carbon emissions from gas fuel consumption",
            "groupChar": "",
            "name": "Gas Fuel",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Carbon emissions from liquid fuel consumption",
            "groupChar": "",
            "name": "Liquid Fuel",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Carbon emissions from solid fuel consumption",
            "groupChar": "",
            "name": "Solid Fuel",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Carbon emissions from cement production",
            "groupChar": "",
            "name": "Cement",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Carbon emissions from gas flaring",
            "groupChar": "",
            "name": "Gas Flaring",
            "type": "number"
          },
          {
            "decimalChar": ".",
            "description": "Per capita carbon emissions (metric tons of carbon; after 1949 only)",
            "groupChar": "",
            "name": "Per Capita",
            "type": "number"
          }
        ]
      },
      "title": "",
      "description": "",
      "alternates": [
        {
          "bytes": 12906,
          "datahub": {
            "type": "source/tabular"
          },
          "dpp:streamedFrom": "https://s3.amazonaws.com/rawstore.datahub.io/09015539c2fc32bb3c4afead7df461b5#.csv",
          "encoding": "utf-8",
          "format": "csv",
          "name": "global",
          "path": "https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global:non-tabular/global.csv",
          "profile": "data-resource",
          "schema": {
            "fields": [
              {
                "description": "Year",
                "format": "any",
                "name": "Year",
                "type": "date"
              },
              {
                "description": "Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)",
                "name": "Total",
                "type": "number"
              },
              {
                "description": "Carbon emissions from gas fuel consumption",
                "name": "Gas Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from liquid fuel consumption",
                "name": "Liquid Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from solid fuel consumption",
                "name": "Solid Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from cement production",
                "name": "Cement",
                "type": "number"
              },
              {
                "description": "Carbon emissions from gas flaring",
                "name": "Gas Flaring",
                "type": "number"
              },
              {
                "description": "Per capita carbon emissions (metric tons of carbon; after 1949 only)",
                "name": "Per Capita",
                "type": "number"
              }
            ]
          }
        },
        {
          "bytes": 46390,
          "datahub": {
            "derivedFrom": [
              "global"
            ],
            "type": "derived/json"
          },
          "dpp:streamedFrom": "https://s3.amazonaws.com/rawstore.datahub.io/09015539c2fc32bb3c4afead7df461b5#.csv",
          "encoding": "utf-8",
          "format": "json",
          "hash": "342d5696d08a9f493b42dcb501b6c768",
          "name": "global_json",
          "path": "https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global:global_json/data/global_json.json",
          "profile": "data-resource",
          "rowcount": 260,
          "schema": {
            "fields": [
              {
                "description": "Year",
                "format": "%Y-%m-%d",
                "name": "Year",
                "type": "date"
              },
              {
                "description": "Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)",
                "name": "Total",
                "type": "number"
              },
              {
                "description": "Carbon emissions from gas fuel consumption",
                "name": "Gas Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from liquid fuel consumption",
                "name": "Liquid Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from solid fuel consumption",
                "name": "Solid Fuel",
                "type": "number"
              },
              {
                "description": "Carbon emissions from cement production",
                "name": "Cement",
                "type": "number"
              },
              {
                "description": "Carbon emissions from gas flaring",
                "name": "Gas Flaring",
                "type": "number"
              },
              {
                "description": "Per capita carbon emissions (metric tons of carbon; after 1949 only)",
                "name": "Per Capita",
                "type": "number"
              }
            ]
          }
        }
      ]
    }
  ],
  "sources": [
    {
      "name": "CDIAC",
      "title": "CDIAC",
      "web": "http://cdiac.esd.ornl.gov/ftp/ndp030/CSV-FILES/global.1751_2010.csv"
    },
    {
      "name": "GitHub repo",
      "title": "GitHub repo",
      "web": "https://github.com/datapackage-examples/transform-co2-fossil-global"
    }
  ],
  "title": "Data Transform examples on Global CO2 Emissions from Fossil Fuels since 1751",
  "views": [
    {
      "id": "graph",
      "label": "Graph",
      "state": {
        "graphType": "lines-and-points",
        "group": "Year",
        "series": [
          "Total",
          "Solid Fuel"
        ]
      },
      "type": "Graph"
    },
    {
      "name": "table-view-from-2000",
      "resources": [
        {
          "name": "global",
          "transform": [
            {
              "expression": "(new Date(data['Year'])).getFullYear() >= 2000",
              "type": "filter"
            }
          ]
        }
      ],
      "specType": "table"
    },
    {
      "name": "table-view-in-percentage",
      "resources": [
        {
          "name": "global",
          "transform": [
            {
              "asFields": [
                "Gas Fuel"
              ],
              "expressions": [
                "data['Gas Fuel'] / data['Total'] * 100 + '%'"
              ],
              "type": "formula"
            }
          ]
        }
      ],
      "specType": "table"
    }
  ]
}

Import into your tool

In order to use Data Package in R follow instructions below:

install.packages("devtools")
library(devtools)
install_github("hadley/readr")
install_github("ropenscilabs/jsonvalidate")
install_github("ropenscilabs/datapkg")

#Load client
library(datapkg)

#Get Data Package
datapackage <- datapkg_read("https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global/latest")

#Package info
print(datapackage)

#Open actual data in RStudio Viewer
View(datapackage$data$"global")

Tested with Python 3.5.2

To generate Pandas data frames based on JSON Table Schema descriptors we have to install jsontableschema-pandas plugin. To load resources from a data package as Pandas data frames use datapackage.push_datapackage function. Storage works as a container for Pandas data frames.

In order to work with Data Packages in Pandas you need to install our packages:

$ pip install datapackage
$ pip install jsontableschema-pandas

To get Data Package run following code:

import datapackage

data_url = "https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global/latest/datapackage.json"

# to load Data Package into storage
storage = datapackage.push_datapackage(data_url, 'pandas')

# to see datasets in this package
storage.buckets

# you can access datasets inside storage, e.g. the first one:
storage[storage.buckets[0]]

In order to work with Data Packages in Python you need to install our packages:

$ pip install datapackage

To get Data Package into your Python environment, run following code:

import datapackage

dp = datapackage.DataPackage('https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global/latest/datapackage.json')

# see metadata
print(dp.descriptor)

# get list of csv files
csvList = [dp.resources[x].descriptor['name'] for x in range(0,len(dp.resources))]
print(csvList) # ["resource name", ...]

# access csv file by the index starting 0
print(dp.resources[0].data)

To use this Data Package in JavaScript, please, follow instructions below:

Install datapackage using npm:

$ npm install [email protected]

Once the package is installed, use code snippet below


const Datapackage = require('datapackage').Datapackage

async function fetchDataPackageAndData(dataPackageIdentifier) {
  const dp = await new Datapackage(dataPackageIdentifier)
  await Promise.all(dp.resources.map(async (resource) => {
    if (resource.descriptor.format === 'geojson') {
      const baseUrl = resource._basePath.replace('/datapackage.json', '')
      const resourceUrl = `${baseUrl}/${resource._descriptor.path}`
      const response = await fetch(resourceUrl)
      resource.descriptor._values = await response.json()
    } else {
      // we assume resource is tabular for now ...
      const table = await resource.table
      // rows are simple arrays -- we can convert to objects elsewhere as needed
      const rowsAsObjects = false
      resource.descriptor._values = await table.read(rowsAsObjects)
    }
  }))

  // see the data package object
  console.dir(dp)

  // data itself is stored in Resource object, e.g. to access first resource:
  console.log(dp.resources[0]._values)

  return dp
}


fetchDataPackageAndData('https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global/latest/datapackage.json');

Our JavaScript is written using ES6 features. We are using node.js v7.4.0 and passing --harmony option to enable ES6:

$ node --harmony index.js

In order to work with Data Packages in SQL you need to install our packages:

$ pip install datapackage
$ pip install jsontableschema-sql
$ pip install sqlalchemy

To import Data Package to your SQLite Database, run following code:

import datapackage
from sqlalchemy import create_engine

data_url = 'https://pkgstore.datahub.io/examples/transform-examples-on-co2-fossil-global/latest/datapackage.json'
engine = create_engine('sqlite:///:memory:')

# to load Data Package into storage
storage = datapackage.push_datapackage(data_url, 'sql', engine=engine)

# to see datasets in this package
storage.buckets

# to execute sql command (assuming data is in "data" folder, name of resource is data and file name is data.csv)
storage._Storage__connection.execute('select * from data__data___data limit 1;').fetchall()

# description of the table columns
storage.describe('data__data___data')
Datapackage.json