waggle/data/README.md at master · waggle-sensor/waggle · GitHub
Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

Latest commit

 

History

History
88 lines (63 loc) · 3.15 KB

README.md

File metadata and controls

88 lines (63 loc) · 3.15 KB
Raw

AoT Data Archive

Historic Data from the Chicago AoT nodes are available for download and analysis. The data will not be updated anymore as the Chicago AoT project has reached EOL.

Data (ontology) is assembled and exported daily, and made available via an archive published every day at midnight CST.

Archive Format

Waggle-based nodes used by several projects, so archives are organized by Waggle ProjectID. Each archive is stored as a single tar file.

Files in the archive use this naming convention:

<ProjectID>.latest.tar

Untarring an archive will create a date-specfic directory:

<ProjectID>.YYYY-MM-DD

Here, YYYY-MM-DD is the UTC date when the tar archive was produced.

Inside the archive directory are the following files:

data.csv.gz	# compressed file of all data values
nodes.csv	# list of nodes in the dataset and their metadata
README.md	# An explaination of the database fields 
sensors.csv	# A list of sensors and their metadata
provenance.csv	# Metadata on the entire dataset archive
offsets.csv     # data.csv.gz file byte offsets

The sensor data is compressed, and must be uncompressed to be used. It will expand roughly 10X in size after being uncompressed.

All data is stored in CSV format.

Available Datasets:

The following AoT Datasets are available -

Usage Example

  1. Download the archive -
$ wget https://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_Chicago.complete.latest.tar
  1. Untar the archive -
$ tar -xf AoT_Chicago.complete.latest.tar
  1. It should produce a directory -
$ ls 
AoT_Chicago.complete.2018-05-22
AoT_Chicago.complete.latest.tar
  1. Explore the README and other meta information -
$ cd AoT_Chicago.complete.2018-05-22
$ ls 
data.csv.gz nodes.csv  provenance.csv   README.md  sensors.csv
  1. When ready, unpack the data archive. The archive decompressess to a CSV text file about 10 times its size -
$ gzip -d --keep data.csv.gz
$ ls 
data.csv data.csv.gz nodes.csv  provenance.csv   README.md  sensors.csv

Working with the Data

We have a few simple tools for to manipulate, extract, create convenient data products and analyze (plot) data. These tools can be found in the Waggle data-tools repository.