Data Transform examples on Global CO2 Emissions from Fossil Fuels since 1751

examples

Files Size Format Created Updated License Source
2 113kB csv zip 9 months ago ODC-PDDL CDIAC GitHub repo
This is an example dataset to demonstrate how data transforms works. In this example, we explain how filtering and applying formula can be done before dataset gets rendered in showcase page. It assumes publisher is already familiar with Data Packages and views specifications (views property in Data read more
Download Developers

Data Files

Download files in this dataset

File Description Size Last changed Download
global 8kB csv (8kB) , json (39kB)
transform-examples-on-co2-fossil-global_zip Compressed versions of dataset. Includes normalized CSV and JSON data with original data and datapackage.json. 12kB zip (12kB)

global  

This is a preview version. There might be more data in the original version.

Field information

Field Name Order Type (Format) Description
Year 1 date (%Y-%m-%d) Year
Total 2 number Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)
Gas Fuel 3 number Carbon emissions from gas fuel consumption
Liquid Fuel 4 number Carbon emissions from liquid fuel consumption
Solid Fuel 5 number Carbon emissions from solid fuel consumption
Cement 6 number Carbon emissions from cement production
Gas Flaring 7 number Carbon emissions from gas flaring
Per Capita 8 number Per capita carbon emissions (metric tons of carbon; after 1949 only)

Integrate this dataset into your favourite tool

Use our data-cli tool designed for data wranglers:

data get https://datahub.io/examples/transform-examples-on-co2-fossil-global
data info examples/transform-examples-on-co2-fossil-global
tree examples/transform-examples-on-co2-fossil-global
# Get a list of dataset's resources
curl -L -s https://datahub.io/examples/transform-examples-on-co2-fossil-global/datapackage.json | grep path

# Get resources

curl -L https://datahub.io/examples/transform-examples-on-co2-fossil-global/r/0.csv

curl -L https://datahub.io/examples/transform-examples-on-co2-fossil-global/r/1.zip

If you are using R here's how to get the data you want quickly loaded:

install.packages("jsonlite", repos="https://cran.rstudio.com/")
library("jsonlite")

json_file <- 'https://datahub.io/examples/transform-examples-on-co2-fossil-global/datapackage.json'
json_data <- fromJSON(paste(readLines(json_file), collapse=""))

# get list of all resources:
print(json_data$resources$name)

# print all tabular data(if exists any)
for(i in 1:length(json_data$resources$datahub$type)){
  if(json_data$resources$datahub$type[i]=='derived/csv'){
    path_to_file = json_data$resources$path[i]
    data <- read.csv(url(path_to_file))
    print(data)
  }
}

Note: You might need to run the script with root permissions if you are running on Linux machine

Install the Frictionless Data data package library and the pandas itself:

pip install datapackage
pip install pandas

Now you can use the datapackage in the Pandas:

import datapackage
import pandas as pd

data_url = 'https://datahub.io/examples/transform-examples-on-co2-fossil-global/datapackage.json'

# to load Data Package into storage
package = datapackage.Package(data_url)

# to load only tabular data
resources = package.resources
for resource in resources:
    if resource.tabular:
        data = pd.read_csv(resource.descriptor['path'])
        print (data)

For Python, first install the `datapackage` library (all the datasets on DataHub are Data Packages):

pip install datapackage

To get Data Package into your Python environment, run following code:

from datapackage import Package

package = Package('https://datahub.io/examples/transform-examples-on-co2-fossil-global/datapackage.json')

# print list of all resources:
print(package.resource_names)

# print processed tabular data (if exists any)
for resource in package.resources:
    if resource.descriptor['datahub']['type'] == 'derived/csv':
        print(resource.read())

If you are using JavaScript, please, follow instructions below:

Install data.js module using npm:

  $ npm install data.js

Once the package is installed, use the following code snippet:

const {Dataset} = require('data.js')

const path = 'https://datahub.io/examples/transform-examples-on-co2-fossil-global/datapackage.json'

// We're using self-invoking function here as we want to use async-await syntax:
;(async () => {
  const dataset = await Dataset.load(path)
  // get list of all resources:
  for (const id in dataset.resources) {
    console.log(dataset.resources[id]._descriptor.name)
  }
  // get all tabular data(if exists any)
  for (const id in dataset.resources) {
    if (dataset.resources[id]._descriptor.format === "csv") {
      const file = dataset.resources[id]
      // Get a raw stream
      const stream = await file.stream()
      // entire file as a buffer (be careful with large files!)
      const buffer = await file.buffer
      // print data
      stream.pipe(process.stdout)
    }
  }
})()

Read me

This is an example dataset to demonstrate how data transforms works. In this example, we explain how filtering and applying formula can be done before dataset gets rendered in showcase page. It assumes publisher is already familiar with Data Packages and views specifications (views property in Data Package specifications).

Transforming data

Data transforms are specified in resources attribute of views property. Each resource is an object that contains following attributes:

  • "name" - name of the resource as a reference.
  • "transform" - array of transforms. Each transform is an object, which properties vary depending on transform type.

Filtering data

Under the graph on the top of this page, you can find a table that displays filtered data. Raw data is displayed in preview section. As you can see we are filtering by year - showing data for 2000 and onwards. This is described in the second view object of views property:

{
  "name": "table-view-from-2000",
  "specType": "table",
  "resources": [
    {
      "name": "global",
      "transform": [
        {
          "type": "filter",
          "expression": "(new Date(data['Year'])).getFullYear() >= 2000"
        }
      ]
    }
  ]
}

where transform property has:

  • "type": "filter" - this way we define the transform to be a filter.
  • "expression" - any JavaScript expression that evaluates to Boolean using some field(s) from the resource. E.g., in our example we are using "Year" field - converting it into JS date object then getting year by using getFullYear() method and filtering by larger or equal sign.

Applying formula

Under the filtered data table, there is another table that displays raw data but with column “Gas Fuels” as a percentage of total figures. JSON representation of it can be found in the third view object:

{
  "name": "table-view-in-percentage",
  "specType": "table",
  "resources": [
    {
      "name": "global",
      "transform": [
        {
          "type": "formula",
          "expressions": [
            "data['Gas Fuel'] / data['Total'] * 100 + '%'"
          ],
          "asFields": ["Gas Fuel"]
        }
      ]
    }
  ]
}

where transform property has:

  • "type": "formula" - defining transform type to be used.
  • "expressions" - list of any JavaScript expressions that will be applied to specified data field.
  • "asFields" - list of field names. This should be used according to list of expressions.

Descriptor for this data package

This is the full datapackage.json of this dataset:

{
  "license": "ODC-PDDL",
  "name": "transform-examples-on-co2-fossil-global",
  "resources": [
    {
      "name": "global",
      "path": "global.csv",
      "format": "csv",
      "schema": {
        "fields": [
          {
            "description": "Year",
            "format": "any",
            "name": "Year",
            "type": "date"
          },
          {
            "description": "Total carbon emissions from fossil fuel consumption and cement production (million metric tons of C)",
            "name": "Total",
            "type": "number"
          },
          {
            "description": "Carbon emissions from gas fuel consumption",
            "name": "Gas Fuel",
            "type": "number"
          },
          {
            "description": "Carbon emissions from liquid fuel consumption",
            "name": "Liquid Fuel",
            "type": "number"
          },
          {
            "description": "Carbon emissions from solid fuel consumption",
            "name": "Solid Fuel",
            "type": "number"
          },
          {
            "description": "Carbon emissions from cement production",
            "name": "Cement",
            "type": "number"
          },
          {
            "description": "Carbon emissions from gas flaring",
            "name": "Gas Flaring",
            "type": "number"
          },
          {
            "description": "Per capita carbon emissions (metric tons of carbon; after 1949 only)",
            "name": "Per Capita",
            "type": "number"
          }
        ]
      }
    }
  ],
  "sources": [
    {
      "name": "CDIAC",
      "web": "http://cdiac.esd.ornl.gov/ftp/ndp030/CSV-FILES/global.1751_2010.csv"
    },
    {
      "name": "GitHub repo",
      "web": "https://github.com/datapackage-examples/transform-co2-fossil-global"
    }
  ],
  "title": "Data Transform examples on Global CO2 Emissions from Fossil Fuels since 1751",
  "views": [
    {
      "id": "graph",
      "label": "Graph",
      "state": {
        "graphType": "lines-and-points",
        "group": "Year",
        "series": [
          "Total",
          "Solid Fuel"
        ]
      },
      "type": "Graph"
    },
    {
      "name": "table-view-from-2000",
      "specType": "table",
      "resources": [
        {
          "name": "global",
          "transform": [
            {
              "type": "filter",
              "expression": "(new Date(data['Year'])).getFullYear() >= 2000"
            }
          ]
        }
      ]
    },
    {
      "name": "table-view-in-percentage",
      "specType": "table",
      "resources": [
        {
          "name": "global",
          "transform": [
            {
              "type": "formula",
              "expressions": [
                "data['Gas Fuel'] / data['Total'] * 100 + '%'"
              ],
              "asFields": ["Gas Fuel"]
            }
          ]
        }
      ]
    }
  ]
}
Datapackage.json