TTN TWIST2018 Dataset

loleg

Files Size Format Created Updated License Source
5 8MB geojson csv zip 3 weeks ago 3 weeks ago
Data from The Things Network (TTN) gateways in Zürich, collected as part of ongoing research and originally provided by Benedikt Hitz-Gamper (Institute of Information Systems, University of Bern) for the TWIST 2018 hackathon. Originally uploaded by @gnz and reformatted as Geospatial Data Package read more
Download Developers

Data Files

Download files in this dataset

File Description Size Last changed Download
ttn_gateways_geojson Gateways map 39kB geojson (39kB)
ttn_gateways Gateways data 23kB csv (23kB) , json (36kB)
ttn_measurements_preview Sample data (preview) 12kB csv (12kB) , json (20kB)
ttn_measurements Sample data (compressed) 8MB zip (8MB)
ttn-twist2018-dataset_zip Compressed versions of dataset. Includes normalized CSV and JSON data with original data and datapackage.json. 8MB zip (8MB)

Gateways map [ttn_gateways_geojson]  

This is a preview version. There might be more data in the original version.

Gateways data [ttn_gateways]  

This is a preview version. There might be more data in the original version.

Field information

Field Name Order Type (Format) Description
device_id 1 integer (default)
eui_id 2 string (default)
platform 3 string (default)
category 4 string (default)
lat 5 number (default)
lng 6 number (default)
altitude 7 number (default)
ETH_dist 8 number (default)

Sample data (preview) [ttn_measurements_preview]  

This is a preview version. There might be more data in the original version.

Field information

Field Name Order Type (Format) Description
id 1 integer (default)
device_id 2 integer (default)
last_online 3 string (default)
rx_ok 4 number (default)
tx_in 5 number (default)
measured_at 6 string (default)

Integrate this dataset into your favourite tool

Use our data-cli tool designed for data wranglers:

data get https://datahub.io/loleg/ttn-twist2018-dataset
data info loleg/ttn-twist2018-dataset
tree loleg/ttn-twist2018-dataset
# Get a list of dataset's resources
curl -L -s https://datahub.io/loleg/ttn-twist2018-dataset/datapackage.json | grep path

# Get resources

curl -L https://datahub.io/loleg/ttn-twist2018-dataset/r/0.geojson

curl -L https://datahub.io/loleg/ttn-twist2018-dataset/r/1.csv

curl -L https://datahub.io/loleg/ttn-twist2018-dataset/r/2.csv

curl -L https://datahub.io/loleg/ttn-twist2018-dataset/r/3.zip

curl -L https://datahub.io/loleg/ttn-twist2018-dataset/r/4.zip

If you are using R here's how to get the data you want quickly loaded:

install.packages("jsonlite", repos="https://cran.rstudio.com/")
library("jsonlite")

json_file <- 'https://datahub.io/loleg/ttn-twist2018-dataset/datapackage.json'
json_data <- fromJSON(paste(readLines(json_file), collapse=""))

# get list of all resources:
print(json_data$resources$name)

# print all tabular data(if exists any)
for(i in 1:length(json_data$resources$datahub$type)){
  if(json_data$resources$datahub$type[i]=='derived/csv'){
    path_to_file = json_data$resources$path[i]
    data <- read.csv(url(path_to_file))
    print(data)
  }
}

Note: You might need to run the script with root permissions if you are running on Linux machine

Install the Frictionless Data data package library and the pandas itself:

pip install datapackage
pip install pandas

Now you can use the datapackage in the Pandas:

import datapackage
import pandas as pd

data_url = 'https://datahub.io/loleg/ttn-twist2018-dataset/datapackage.json'

# to load Data Package into storage
package = datapackage.Package(data_url)

# to load only tabular data
resources = package.resources
for resource in resources:
    if resource.tabular:
        data = pd.read_csv(resource.descriptor['path'])
        print (data)

For Python, first install the `datapackage` library (all the datasets on DataHub are Data Packages):

pip install datapackage

To get Data Package into your Python environment, run following code:

from datapackage import Package

package = Package('https://datahub.io/loleg/ttn-twist2018-dataset/datapackage.json')

# print list of all resources:
print(package.resource_names)

# print processed tabular data (if exists any)
for resource in package.resources:
    if resource.descriptor['datahub']['type'] == 'derived/csv':
        print(resource.read())

If you are using JavaScript, please, follow instructions below:

Install data.js module using npm:

  $ npm install data.js

Once the package is installed, use the following code snippet:

const {Dataset} = require('data.js')

const path = 'https://datahub.io/loleg/ttn-twist2018-dataset/datapackage.json'

// We're using self-invoking function here as we want to use async-await syntax:
;(async () => {
  const dataset = await Dataset.load(path)
  // get list of all resources:
  for (const id in dataset.resources) {
    console.log(dataset.resources[id]._descriptor.name)
  }
  // get all tabular data(if exists any)
  for (const id in dataset.resources) {
    if (dataset.resources[id]._descriptor.format === "csv") {
      const file = dataset.resources[id]
      // Get a raw stream
      const stream = await file.stream()
      // entire file as a buffer (be careful with large files!)
      const buffer = await file.buffer
      // print data
      stream.pipe(process.stdout)
    }
  }
})()

Read me

Data from The Things Network (TTN) gateways in Zürich, collected as part of ongoing research and originally provided by Benedikt Hitz-Gamper (Institute of Information Systems, University of Bern) for the TWIST 2018 hackathon.

Originally uploaded by @gnz and reformatted as Geospatial Data Package by @loleg

Data files

File ttn_gateways.csv (134 rows)

  • device_id: unique key for device_id in table ttn_measurements (this is an internal key, not connected with TTN IDs)
  • eui_id: id used by TTN to uniquely identify gateways
  • platform: make/model of gateway if provided
  • category: clustered value for platform
  • lat/lng: location of gateway if provided
  • altitude: altitute of gateway if provided
  • ETH_dist: distance from gateway to ETH main building (selection criteria for gateways is < 20km)

File ttn_measurements.csv (750926 rows)

  • id: unique id
  • device_id: key for device_id in table ttn_gateways
  • last_online: last handshake of gateway with TTN backend
  • rx_ok: number of packets received by this gateway (counter is never reset)
  • tx_in: number of packets sent by this gateway (counter is never reset)
  • measured_at: time of measurement

To decide whether a gateway is online, verify the difference of measured_at - last_online. If difference is smaller than 30 to 90 seconds, the gateway was online at this time.

Note: Somethimes last_online is greater than measured_at, this has to do with the fact that the values get actually stored in the database after measured_at and sometimes even a newer last_online is available.

Related datasets

Credit

Data part of ongoing research, provided by Benedikt Hitz-Gamper, Institute of Information Systems, University of Bern, [email protected]

License

The content in this repository is licensed under the CC-BY-SA 4.0 License (see the LICENSE file for details) unless stated otherwise.

License: CC BY-SA 4.0

Datapackage.json