Oracle Data

The Helium Network operates Oracles that perform various tasks backing Proof-of-Coverage and data transfer for the LoRaWAN (IOT), Mobile, and any future networks. These Oracles provide extensive logs for analysis of the Helium subnetworks.

The Helium Oracles handle all data about the network, except for transactions handled by the Helium Program Libraries on Solana.

Accessing Oracle Data

Data is directly accessible from the foundation-poc-data-requester-pays bucket in AWS us-west-2 under the top-level data category prefixes listed below. All data provided in the S3 bucket is made available on a basis of 'Requestor Pays', meaning you - the requester - will pay for the data transfer and AWS API costs incurred from the data requests.

For more information on accessing data from an AWS requester pays bucket, refer to Downloading objects in Requester Pays buckets.

Top-Level Data Category Prefixes
foundation-iot-entropy
foundation-iot-ingest
foundation-iot-packet-ingest
foundation-iot-packet-verifier
foundation-iot-verified-rewards
foundation-mobile-ingest
foundation-mobile-packet-verifier
foundation-mobile-price
foundation-mobile-verified

Under each top-level data category prefix, the related PoC data files are provided in the prefix structure noted below:

s3://foundation-poc-data-requester-pays/<top-level-data-category-prefix>/<file_prefix>.<unix_ms_timestamp>.gz

Again, as the foundation-poc-data-requester-pays bucket is 'Requestor Pays', be mindful that AWS will charge you for:

1. Storage costs on any destination buckets
2. Egress fees if the data leaves us-west-2
3. PUT, COPY, POST, and LIST requests

To that end, keeping data in us-west-2 and being pragmatic about which AWS S3 access patterns are used will greatly reduce the cost burden for accessing the PoC data.

Regarding access patterns, one approach to minimize the quantity of S3 API calls over time - and save money - is to use the aws s3api list-objects-v2 command and specify the --start-after and --prefix parameters. In particular, the aws s3api list-objects-v2 command can be invoked iteravtively over time with the --start-after parameter set to the top-level data category prefix + filename of the last file received and the --prefix parameter set to the top-level data category + file prefix of desired file, as indicated below:

aws s3api list-objects-v2 --bucket foundation-poc-data-requester-pays  --start-after <top-level-data-category-prefix/filename> --prefix <top-level-data-category-prefix/file_prefix> --request-payer requester

For example, if you were to execute the aws s3api list-objects-v2 command against a foo prefix of an example bucket, and the most recent file you received was bar.1681917512317.gz, in a subsequent invocation of aws s3api list-objects-v2 you could set the --start-after flag to foo/bar.1681917512317.gz and the --prefix flag to foo/bar to return a list of bar type files subsequent to bar.1681917512317.gz. In doing so, your total API calls in the second invocation of aws s3api list-objects-v2 would be limited to only the files subsequent to bar.1681917512317.gz, rather than for all files in the bucket. From there, you can iterate over the returned list of files and perform additional S3 API commands such as aws s3api copy-object to copy files based on the file list.

Oracle Data File Format

The raw PoC data stored in the foundation-poc-data-requester-pays bucket is in encoded protobuf format in .gz compressed files. The table below provides the top-level data category prefixes, the file types comprised within each data category prefix, the file prefixes beginning each filename, the filename patterns for each of the file types, and links to the corresponding protobuf used to
decode the data.

Category	File Type	File Prefix	Filename	Protobuf Definition
foundation-iot-entropy	EntropyReport	entropy_report	entropy_report.<unix_ms_timestamp>.gz	entropy.proto (Line 5: entropy_report_v1)
foundation-iot-ingest	IotBeaconIngestReport	iot_beacon_ingest_report	iot_beacon_ingest_report.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 97: lora_beacon_ingest_report_v1)
foundation-iot-ingest	IotWitnessIngestReport	iot_witness_ingest_report	iot_witness_ingest_report.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 104: lora_witness_ingest_report_v1)
foundation-iot-packet-ingest	PacketReport	packetreport	packetreport.<unix_ms_timestamp>.gz	service/packet_router.proto (Line 8: packet_router_packet_report_v1)
foundation-iot-packet-verifier	InvalidPacket	invalid_packet	invalid_packet.<unix_ms_timestamp>.gz	service/packet_verifier.proto (Line 13: invalid_packet)
foundation-iot-packet-verifier	IotValidPacket	iot_valid_packet	iot_valid_packet.<unix_ms_timestamp>.gz	service/packet_verifier.proto (Line 5: valid_packet)
foundation-iot-verified-rewards	IotRewardShare	iot_reward_share	iot_reward_share.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 239: iot_reward_share)
foundation-iot-verified-rewards	IotPoc	iot_poc	iot_poc.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 216: lora_poc_v1)
foundation-iot-verified-rewards	IotInvalidBeaconReport	iot_invalid_beacon	iot_invalid_beacon.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 153: lora_invalid_beacon_report_v1)
foundation-iot-verified-rewards	IotInvalidWitnessReport	iot_invalid_witness	iot_invalid_witness.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 173: lora_invalid_witness_report_v1)
foundation-iot-verified-rewards	GatewayRewardShare	gateway_reward_share(Deprecated 4/18/23)	gateway_reward_share.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 171: gateway_reward_share)
foundation-iot-verified-rewards	RewardManifest	reward_manifest	reward_manifest.<unix_ms_timestamp>.gz	reward_manifest.proto (Line 5: reward_manifest)
foundation-iot-verified-rewards	NonRewardablePacket	non_rewardable_packet	non_rewardable_packet.<unix_ms_timestamp>.gz	service/poc_lora.proto (Line 47: non_rewardable_packet)
foundation-mobile-ingest	CellHeartbeatIngestReport	cell_heartbeat	cell_heartbeat.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 65: cell_heartbeat_ingest_report_v1)
foundation-mobile-ingest	CellSpeedtestIngestReport	cell_speedtest	cell_speedtest.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 25: speedtest_ingest_report_v1)
foundation-mobile-ingest	DataTransferSessionIngestReport	data_transfer_session_ingest_report	data_transfer_session_ingest_report.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 366: data_transfer_session_ingest_report_v1)
foundation-mobile-ingest	Heartbeat	heartbeat	heartbeat.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 181: heartbeat)
foundation-mobile-ingest	SpeedtestIngestReport	speedtest_report	speedtest_report.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 25: speedtest_ingest_report_v1)
foundation-mobile-ingest	SubscriberLocationIngestReport	subscriber_location_report	subscriber_location_report.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 92: subscriber_location_ingest_report_v1)
foundation-mobile-packet-verifier	InvalidDataTransferSessionIngestReport	invalid_data_transfer_session_ingest_report	invalid_data_transfer_session_ingest_report.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 372: invalid_data_transfer_ingest_report_v1)
foundation-mobile-packet-verifier	ValidDataTransferSession	valid_data_transfer_session	valid_data_transfer_session.<unix_ms_timestamp>.gz	service/packet_verifier.proto (Line 24: valid_data_transfer_session)
foundation-mobile-price	PriceReport	price_report	price_report.<unix_ms_timestamp>.gz	price_report.proto (Line 7: price_report_v1)
foundation-mobile-verified	ValidatedHeartbeat	validated_heartbeat	validated_heartbeat.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 181: heartbeat)
foundation-mobile-verified	SpeedtestAvg	speedtest_avg	speedtest_avg.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line L226: speedtest_avg)
foundation-mobile-verified	VerifiedSpeedtest	verified_speedtest	verified_speedtest.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line L36: verified_speedtest)
foundation-mobile-verified	VerifiedSubscriberLocationIngestReport	verified_subscriber_location_report	verified_subscriber_location_report.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 140: verified_subscriber_location_ingest_report_v1)
foundation-mobile-verified	RadioRewardShare	radio_reward_share	radio_reward_share.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 247: radio_reward_share)
foundation-mobile-verified	MobileRewardShare	mobile_reward_share	mobile_reward_share.<unix_ms_timestamp>.gz	service/poc_mobile.proto (Line 295: mobile_reward_share)
foundation-mobile-verified	RewardManifest	reward_manifest	reward_manifest.<unix_ms_timestamp>.gz	reward_manifest.proto (Line 5: reward_manifest)
foundation-mobile-verified	SubnetworkRewards	subnetwork_rewards	subnetwork_rewards.<unix_ms_timestamp>.gz	blockchain_txn_subnetwork_rewards_v1.proto (Line 11: subnetwork_rewards)
foundation-mobile-verified	InvalidShares	invalid_shares	invalid_shares.<unix_ms_timestamp>.gz
foundation-mobile-verified	Shares	shares	shares.<unix_ms_timestamp>.gz

Reading Bytestream Data

To store the massive amount of data generated by the Oracles, the output is stored as zipped bytestreams (one bytestream per .gz file). To 'unpack' this data, you'll first need the associated Protobuf from github.com/helium/proto (also listed above).

Once uncompressed, the file is a series of 4-byte prefix length encoded binary followed by the associated message. More concretely, you'll need to repeatedly read the first 4 bytes and decode that as a big-endian u32. Then use that u32 number as the number of subsequent bytes to read. Decode those subsequent bytes using the associated proto definition. Repeat this process until you've read the entire file.

Important Note: Simply trying to pass the contents of the uncompressed files into the protobuf decoder will fail.

Javascript Example

// poc_mobile.proto
// copied from https://github.com/helium/proto/blob/master/src/service/poc_mobile.proto#L31

syntax = "proto3";

package helium.poc_mobile;

message cell_heartbeat_req_v1 {
  // Public key of the hotspot
  bytes pub_key = 1;
  string hotspot_type = 2;
  uint32 cell_id = 3;
  // Timestamp of heartbeat in seconds from unix epoch
  uint64 timestamp = 4;
  double lat = 5;
  double lon = 6;
  bool operation_mode = 7;
  string cbsd_category = 8;
  string cbsd_id = 9;

  bytes signature = 10;
}

// oracle-data-reader-example.js

// assumes you've downloaded and unzipped the below file
const fileBuffer = fs.readFileSync('./cell_heartbeat.1667919103443')

protobuf.load('mobile.proto', function (err, root) {
  if (err) throw err

  // Obtain a message type
  var CellHeartbeat = root.lookupType('helium.poc_mobile.cell_heartbeat_req_v1')

  let offset = 0
  while (offset < fileBuffer.length) {
    // the first 4 bytes tell you how long the message is
    const messageLength = fileBuffer.readInt32BE(offset)
    const bufferMessage = fileBuffer.subarray(offset + 4, offset + messageLength + 4)
    try {
      var decodedMessage = CellHeartbeat.decode(bufferMessage)
      console.log(decodedMessage)
    } catch (e) {
      console.log('error', e)
    }
    offset += messageLength + 4
  }
})

Rust Example

This leverages existing Rust dependencies used by the Oracles.

# Cargo.toml
[package]
name = "oracle-data-reader-example"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = {version = "1", features = ["backtrace"]}
file-store = {git = "https://github.com/helium/oracles", branch = "main"}
futures = "*"
serde =  {version = "1", features=["derive"]}
serde_json = "1"
tokio = { version = "1", default-features = false, features = [
    "fs",
    "macros",
    "signal",
    "rt-multi-thread",
    "rt",
    "process",
    "time"
] }
tokio-util = "0"

# main.rs
use anyhow::Result;
use file_store::{heartbeat::CellHeartbeat, traits::MsgDecode, file_source};
use futures::{ StreamExt};
use serde_json::json;

fn print_json<T: ?Sized + serde::Serialize>(value: &T) -> Result<()> {
    println!("{}", serde_json::to_string_pretty(value)?);
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    // assumes you have cell_heartbeat.1667919103443.gz in folder
    let mut file_stream = file_source::source(["cell_heartbeat.1667919103443.gz"]);

    while let Some(result) = file_stream.next().await {
        let msg = result?;
        let heartbeat = CellHeartbeat::decode(msg)?;
        print_json(&json!(heartbeat))?;
    }
    Ok(())
}

A more fleshed-out example can be found in this sample oracle observer repo.

Service Providers

Several Helium Ecosystem companies are building interfaces around Oracle Data. These services range from APIs to Oracle Data Explorers

• Hotspotty
• HeliumGeek
• Helium Deploy