Oracle Data
The Helium Network operates Oracles that perform various tasks backing Proof of Coverage and data transfer for the LoRaWAN (IoT), Mobile, and any future networks. These Oracles provide extensive logs for the purposes of analysis of the Helium subDAOs.
The Helium Oracles handle all data about the network with the exception of transactions handled by the Helium Program Libraries on Solana.
Accessing Oracle Data
AWS S3 Buckets
Data is directly accessible from the foundation-poc-data-requester-pays bucket in AWS us-west-2
under the top level data category prefixes listed below. All data provided in the S3 bucket is made
available on a basis of 'Requestor Pays' meaning you, the requester, will pay for the data transfer
and API costs incurred from the data requests. For more information on accessing data from a
requester pays bucket, refer to
Downloading objects in Requester Pays buckets.
| Top Level Data Category Prefixes |
|---|
foundation-iot-entropy |
foundation-iot-ingest |
foundation-iot-packet-ingest |
foundation-iot-packet-verifier |
foundation-iot-verified-rewards |
foundation-mobile-ingest |
foundation-mobile-verified |
Under each top level data category prefix, the relating PoC data files are provided in a flattened structure in the format provided below:
s3://foundation-poc-data-requester-pays/<top-level-data-category-prefix>/<filename>.gz
Again, as the foundation-poc-data-requester-pays bucket is 'Requestor Pays', you should be mindful
that AWS will charge you for:
- Storage costs on any destination buckets
- Egress fees if the data leaves
us-west-2 PUT,COPY,POST,LISTrequests
To that end, keeping data in us-west-2 and being pragmatic about which AWS S3 access patterns you
use will greatly reduce your cost burden for accessing the PoC data.
In regard to access patterns, one approach that you can take to minimize the quantity of S3 API
calls over time - and save money - is to use the aws s3api list-objects-v2 command and specify the
--start-after parameter with the filename of the last file you received, as indicated below:
aws s3api list-objects-v2 --bucket foundation-poc-data-requester-pays --prefix <top-level-data-category-prefix> --start-after <filename> --request-payer requester
For example, if you were to execute the aws s3api list-objects-v2 command and the last / most
recent file you received was example.gz, in a subsequent call to aws s3api list-objects-v2 you
could add the --start-after example.gz flag to return a list of files subsequent to example.gz.
In doing so, your total API calls in the second invocation of aws s3api list-objects-v2 would be
limited to only the files subsequent to example.gz, rather than for all files in the bucket. From
there, you can iterate over the returned list of files and perform additional S3 API commands such
as aws s3api copy-object to copy files based on the file list.
The raw PoC data stored in the foundation-poc-data-requester-pays bucket is in encoded protobuf
format in gz compressed files. The table below provides the top level data category prefixes, the
file types comprised within each data category prefix, the filename pattern for each file type, and
provides a link to the corresponding protobuf used to decode the data.
| Category | File Type | Pattern | Protobuf |
|---|---|---|---|
| foundation-iot-entropy | Entropy | entropy.* | Proto |
| foundation-iot-ingest | IotBeaconIngestReport | iot_beacon_ingest_report.* | Proto |
| foundation-iot-ingest | IotWitnessIngestReport | iot_witness_ingest_report.* | Proto |
| foundation-iot-packet-ingest | PacketReport | packetreport.* | Proto |
| foundation-iot-packet-verifier | Coming soon | ||
| foundation-iot-verified-rewards | IotPoc | iot_poc.* | Proto |
| foundation-iot-verified-rewards | IotInvalidBeaconReport | iot_invalid_beacon.* | Proto |
| foundation-iot-verified-rewards | IotInvalidWitnessReport | iot_invalid_witness.* | Proto |
| foundation-iot-verified-rewards | GatewayRewardShare | gateway_reward_share.* | Proto |
| foundation-iot-verified-rewards | RewardManifest | reward_manifest.* | Proto |
| foundation-mobile-ingest | CellHeartbeatIngestReport | heartbeat_report.* | Proto |
| foundation-mobile-ingest | CellSpeedtestIngestReport | speedtest_report.* | Proto |
| foundation-mobile-ingest | DataTransferSessionIngestReport | data_transfer_session_ingest_report.* | Proto |
| foundation-mobile-verified | ValidatedHeartbeat | validated_heartbeat.* | Proto |
| foundation-mobile-verified | SpeedtestAvg | speedtest_avg.* | Proto |
| foundation-mobile-verified | RadioRewardShare | radio_reward_share.* | Proto |
| foundation-mobile-verified | RewardManifest | reward_manifest.* | Proto |
Arweave
Planned: In the interest of providing a decentralized option for data access all Oracle data is additionally replicated to Arweave. This provides a data access option that does not require egress fees. Arweave implementation will be accompanied by relevant access patterns.
Reading Bytestream Data
In order to store the massive amount of data generated by the Oracles, the output is stored as zipped bytestreams. In order to 'unpack' this data, you'll need the associated protobuf from github.com/helium/proto.
Service Providers
Several Helium ecosystem companies are building interfaces around Oracle data. These services range from APIs to Oracle data explorers