Helium Oracle Data

The Helium Network operates Oracles that perform various tasks backing Proof-of-Coverage and data transfer for the LoRaWAN (IOT), Mobile, and any future networks. These Oracles provide extensive logs for analysis of the Helium subnetworks.

The Helium Oracles handle all data about the network, except for transactions handled by the Helium Program Libraries on Solana.

Accessing Oracle Data

Data is directly accessible from the foundation-poc-data-requester-pays bucket in AWS us-west-2 under the top-level data category prefixes listed below. All data in the S3 bucket uses 'Requestor Pays', meaning you pay for data transfer and AWS API costs. Amazon will charge you for:

Storage costs on any destination buckets
Egress fees if the data leaves us-west-2
PUT, COPY, POST, and LIST requests

For more information on accessing data from an AWS requester pays bucket, refer to the Downloading objects in Requester Pays buckets guide provided by AWS.

Top-Level Data Category Prefixes
`foundation-iot-entropy`
`foundation-iot-ingest`
`foundation-iot-packet-ingest`
`foundation-iot-packet-verifier`
`foundation-iot-verified-rewards`
`foundation-mobile-ingest`
`foundation-mobile-packet-verifier`
`foundation-mobile-price`
`foundation-mobile-verified`

Under each top-level data category prefix, the related PoC data files are provided in the prefix structure noted below:

s3://foundation-poc-data-requester-pays/<top-level-data-category-prefix>/<file_prefix>.<unix_ms_timestamp>.gz

When working with this bucket, keeping data in us-west-2 and being pragmatic about which AWS S3 access patterns are used will greatly reduce the cost burden for accessing the PoC data.

Regarding access patterns, one approach to minimize the quantity of S3 API calls over time - and save money - is to use the aws s3api list-objects-v2 command and specify the --start-after and --prefix parameters. In particular, the aws s3api list-objects-v2 command can be invoked iteratively over time with the --start-after parameter set to the top-level data category prefix + filename of the last file received and the --prefix parameter set to the top-level data category + file prefix of desired file, as indicated below:

aws s3api list-objects-v2 --bucket foundation-poc-data-requester-pays  --start-after <top-level-data-category-prefix/filename> --prefix <top-level-data-category-prefix/file_prefix> --request-payer requester

For example, if you were to execute the aws s3api list-objects-v2 command against a foo prefix of an example bucket, and the most recent file you received was bar.1681917512317.gz, in a subsequent invocation of aws s3api list-objects-v2 you could set the --start-after flag to foo/bar.1681917512317.gz and the --prefix flag to foo/bar to return a list of bar type files subsequent to bar.1681917512317.gz. In doing so, your total API calls in the second invocation of aws s3api list-objects-v2 would be limited to only the files subsequent to bar.1681917512317.gz, rather than for all files in the bucket. From there, you can iterate over the returned list of files and perform additional S3 API commands such as aws s3api copy-object to copy files based on the file list.

Oracle Data File Format

The raw PoC data stored in the foundation-poc-data-requester-pays bucket is in encoded protobuf format in .gz compressed files. The table below provides the top-level data category prefixes, the file types comprised within each data category prefix, the file prefixes beginning each filename, the filename patterns for each of the file types, and links to the corresponding protobuf used to
decode the data.

Category	File Type	File Prefix	Filename	Protobuf Definition
`foundation-iot-entropy`	`EntropyReport`	`entropy_report`	`entropy_report.<unix_ms_timestamp>.gz`	`entropy.proto` (Line 5: `entropy_report_v1`)
`foundation-iot-ingest`	`IotBeaconIngestReport`	`iot_beacon_ingest_report`	`iot_beacon_ingest_report.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 97: `lora_beacon_ingest_report_v1`)
`foundation-iot-ingest`	`IotWitnessIngestReport`	`iot_witness_ingest_report`	`iot_witness_ingest_report.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 104: `lora_witness_ingest_report_v1`)
`foundation-iot-packet-ingest`	`PacketReport`	`packetreport`	`packetreport.<unix_ms_timestamp>.gz`	`service/packet_router.proto` (Line 8: `packet_router_packet_report_v1`)
`foundation-iot-packet-verifier`	`InvalidPacket`	`invalid_packet`	`invalid_packet.<unix_ms_timestamp>.gz`	`service/packet_verifier.proto` (Line 13: `invalid_packet`)
`foundation-iot-packet-verifier`	`IotValidPacket`	`iot_valid_packet`	`iot_valid_packet.<unix_ms_timestamp>.gz`	`service/packet_verifier.proto` (Line 5: `valid_packet`)
`foundation-iot-verified-rewards`	`IotRewardShare`	`iot_reward_share`	`iot_reward_share.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 239: `iot_reward_share`)
`foundation-iot-verified-rewards`	`IotPoc`	`iot_poc`	`iot_poc.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 216: `lora_poc_v1`)
`foundation-iot-verified-rewards`	`IotInvalidBeaconReport`	`iot_invalid_beacon`	`iot_invalid_beacon.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 153: `lora_invalid_beacon_report_v1`)
`foundation-iot-verified-rewards`	`IotInvalidWitnessReport`	`iot_invalid_witness`	`iot_invalid_witness.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 173: `lora_invalid_witness_report_v1`)
`foundation-iot-verified-rewards`	`GatewayRewardShare`	`gateway_reward_share`(Deprecated 4/18/23)	`gateway_reward_share.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 171: `gateway_reward_share`)
`foundation-iot-verified-rewards`	`RewardManifest`	`reward_manifest`	`reward_manifest.<unix_ms_timestamp>.gz`	`reward_manifest.proto` (Line 5: `reward_manifest`)
`foundation-iot-verified-rewards`	`NonRewardablePacket`	`non_rewardable_packet`	`non_rewardable_packet.<unix_ms_timestamp>.gz`	`service/poc_lora.proto` (Line 47: `non_rewardable_packet`)
`foundation-mobile-ingest`	`CellHeartbeatIngestReport`	`cell_heartbeat`	`cell_heartbeat.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 65: `cell_heartbeat_ingest_report_v1`)
`foundation-mobile-ingest`	`CellSpeedtestIngestReport`	`cell_speedtest`	`cell_speedtest.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 25: `speedtest_ingest_report_v1`)
`foundation-mobile-ingest`	`DataTransferSessionIngestReport`	`data_transfer_session_ingest_report`	`data_transfer_session_ingest_report.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 366: `data_transfer_session_ingest_report_v1`)
`foundation-mobile-ingest`	`Heartbeat`	`heartbeat`	`heartbeat.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 181: `heartbeat`)
`foundation-mobile-ingest`	`SpeedtestIngestReport`	`speedtest_report`	`speedtest_report.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 25: `speedtest_ingest_report_v1`)
`foundation-mobile-ingest`	`SubscriberLocationIngestReport`	`subscriber_location_report`	`subscriber_location_report.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 92: `subscriber_location_ingest_report_v1`)
`foundation-mobile-packet-verifier`	`InvalidDataTransferSessionIngestReport`	`invalid_data_transfer_session_ingest_report`	`invalid_data_transfer_session_ingest_report.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 372: `invalid_data_transfer_ingest_report_v1`)
`foundation-mobile-packet-verifier`	`ValidDataTransferSession`	`valid_data_transfer_session`	`valid_data_transfer_session.<unix_ms_timestamp>.gz`	`service/packet_verifier.proto` (Line 24: `valid_data_transfer_session`)
`foundation-mobile-price`	`PriceReport`	`price_report`	`price_report.<unix_ms_timestamp>.gz`	`price_report.proto` (Line 7: `price_report_v1`)
`foundation-mobile-verified`	`ValidatedHeartbeat`	`validated_heartbeat`	`validated_heartbeat.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 181: `heartbeat`)
`foundation-mobile-verified`	`SpeedtestAvg`	`speedtest_avg`	`speedtest_avg.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line L226: `speedtest_avg`)
`foundation-mobile-verified`	`VerifiedSpeedtest`	`verified_speedtest`	`verified_speedtest.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line L36: `verified_speedtest`)
`foundation-mobile-verified`	`VerifiedSubscriberLocationIngestReport`	`verified_subscriber_location_report`	`verified_subscriber_location_report.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 140: `verified_subscriber_location_ingest_report_v1`)
`foundation-mobile-verified`	`RadioRewardShare`	`radio_reward_share`	`radio_reward_share.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 247: `radio_reward_share`)
`foundation-mobile-verified`	`MobileRewardShare`	`mobile_reward_share`	`mobile_reward_share.<unix_ms_timestamp>.gz`	`service/poc_mobile.proto` (Line 295: `mobile_reward_share`)
`foundation-mobile-verified`	`RewardManifest`	`reward_manifest`	`reward_manifest.<unix_ms_timestamp>.gz`	`reward_manifest.proto` (Line 5: `reward_manifest`)
`foundation-mobile-verified`	`SubnetworkRewards`	`subnetwork_rewards`	`subnetwork_rewards.<unix_ms_timestamp>.gz`	`blockchain_txn_subnetwork_rewards_v1.proto` (Line 11: `subnetwork_rewards`)
`foundation-mobile-verified`	`InvalidShares`	`invalid_shares`	`invalid_shares.<unix_ms_timestamp>.gz`
`foundation-mobile-verified`	`Shares`	`shares`	`shares.<unix_ms_timestamp>.gz`

Reading Bytestream Data

To store the massive amount of data generated by the Oracles, the output is stored as zipped bytestreams (one bytestream per .gz file). To 'unpack' this data, you'll first need the associated Protobuf from github.com/helium/proto (also listed above).

Once uncompressed, the file is a series of 4-byte prefix length encoded binary followed by the associated message. More concretely, you'll need to repeatedly read the first 4 bytes and decode that as a big-endian u32. Then use that u32 number as the number of subsequent bytes to read. Decode those subsequent bytes using the associated proto definition. Repeat this process until you've read the entire file.

Important Note: Simply trying to pass the contents of the uncompressed files into the protobuf decoder will fail.

Javascript Example

// poc_mobile.proto
// copied from https://github.com/helium/proto/blob/master/src/service/poc_mobile.proto#L31

syntax = "proto3";

package helium.poc_mobile;

message cell_heartbeat_req_v1 {
  // Public key of the hotspot
  bytes pub_key = 1;
  string hotspot_type = 2;
  uint32 cell_id = 3;
  // Timestamp of heartbeat in seconds from unix epoch
  uint64 timestamp = 4;
  double lat = 5;
  double lon = 6;
  bool operation_mode = 7;
  string cbsd_category = 8;
  string cbsd_id = 9;

  bytes signature = 10;
}

// oracle-data-reader-example.js

// assumes you've downloaded and unzipped the below file
const fileBuffer = fs.readFileSync('./cell_heartbeat.1667919103443')

protobuf.load('mobile.proto', function (err, root) {
  if (err) throw err

  // Obtain a message type
  var CellHeartbeat = root.lookupType('helium.poc_mobile.cell_heartbeat_req_v1')

  let offset = 0
  while (offset < fileBuffer.length) {
    // the first 4 bytes tell you how long the message is
    const messageLength = fileBuffer.readInt32BE(offset)
    const bufferMessage = fileBuffer.subarray(offset + 4, offset + messageLength + 4)
    try {
      var decodedMessage = CellHeartbeat.decode(bufferMessage)
      console.log(decodedMessage)
    } catch (e) {
      console.log('error', e)
    }
    offset += messageLength + 4
  }
})

Rust Example

This leverages existing Rust dependencies used by the Oracles.

# Cargo.toml
[package]
name = "oracle-data-reader-example"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = {version = "1", features = ["backtrace"]}
file-store = {git = "https://github.com/helium/oracles", branch = "main"}
futures = "*"
serde =  {version = "1", features=["derive"]}
serde_json = "1"
tokio = { version = "1", default-features = false, features = [
    "fs",
    "macros",
    "signal",
    "rt-multi-thread",
    "rt",
    "process",
    "time"
] }
tokio-util = "0"

# main.rs
use anyhow::Result;
use file_store::{heartbeat::CellHeartbeat, traits::MsgDecode, file_source};
use futures::{ StreamExt};
use serde_json::json;

fn print_json<T: ?Sized + serde::Serialize>(value: &T) -> Result<()> {
    println!("{}", serde_json::to_string_pretty(value)?);
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    // assumes you have cell_heartbeat.1667919103443.gz in folder
    let mut file_stream = file_source::source(["cell_heartbeat.1667919103443.gz"]);

    while let Some(result) = file_stream.next().await {
        let msg = result?;
        let heartbeat = CellHeartbeat::decode(msg)?;
        print_json(&json!(heartbeat))?;
    }
    Ok(())
}

A more fleshed-out example can be found in this sample oracle observer repo.

Accessing Oracle Data​

Oracle Data File Format​

Reading Bytestream Data​

Javascript Example​

Rust Example​

Accessing Oracle Data

Oracle Data File Format

Reading Bytestream Data

Javascript Example

Rust Example