On 2023-04-18, a new Config Service release was deployed as part of the Oracle 1.0.0 release, which included a more rigorous request signature verification. This caused a protobuf mismatch with Packet Router and Helium Router.
This latest update adds more behind the scenes support for the Solana migration, including transactions, adding, updating, and viewing Hotspots on Solana L1, and swap support.
In addition, 1.6.0 fixes a bug where users could not sign into Crowdspot and fixes the display of HNT price on iOS widgets.
The Nova developers have tagged a new 2022.12.13.0 Hotspot release for Original Helium Hotspots.
This release prepares Hotspots for off-chain proof-of-coverage by updating the automatic beacon rate
to every 6 hours, adds "jitter" to the beacon rate such that in the event all Hotspots come online
at the same time, the PoC system is not overwhelmed with PoC.
Prepare for the Helium migration to the Solana blockchain! This app update makes it easier for "blue app" users to successfully migrate to the Wallet app to manage their tokens. For those already on the Wallet app, migration will be easy. Just make sure your app is updated to the latest versions and your tokens will automatically be updated to use Solana.
On Wednesday, 2022-11-16, at approximately 22:06 UTC, Validators stopped producing blocks at block
1,619,546, and a chain halt occurred. Normal chain operations resumed 18 hours on 2022-11-17 at
approximately 15:57 UTC.
The Nova developers have tagged a new 2022.11.10.0 firmware release for Original Helium Hotspots.
This release updates the regional frequency plan to their latest versions as well as implements the
logic required to digest band plan change.
On Wednesday, 2022-10-26, at approximately 1:57 AM UTC, Validators stopped producing blocks at block
1,587,313, and a chain halt occurred. Consequently, two days later, at 3:57 PM UTC, a second chain
halt occurred at block 1,589,496. Normal chain operations resumed approximately 32 hours later at
2022-10-30 1:32 AM UTC.
The Wallet app now supports burn transactions from Helium Console. To burn Data Credits for packet transfer, simply tap the "Send" button and the QR code in the top left corner and scan the QR code.
The core developers have tagged a new 2022.10.28.0 miner release. This release adds exponential
back-off for Hotspots when requesting ledger information from Validators. This also ensures timely
back-off when block times are too high and avoids too many Hotspots attempting to connect to a
Validator. The pull request specifically addresses the blockchain outages over the past week from a
Hotspot's perspective.
On Wednesday 2022-10-19 at approximately 2:20 AM UTC Validators stopped producing blocks at block
1,575,935 and a chain halt occurred. Consequently, at 5:55 AM UTC a second chain halt occurred at
block 1,575,955. Normal chain operations resumed approximately 5.5 hours later at 7:49 AM.
The UI has been cleaned to show a chart based on $HNT oracle price, made voting more intuitive, and improved payments on subtokens.
Do you have a CLI Wallet? Do you wish you had a user interface to use instead? Now you can!
The core developers have tagged a new 2022.08.17.1 miner release. This release disables
peer-to-peer activity when the Hotspot is no longer syncing the chain. This improves overall
performance for the peer-to-peer network for other participants and stability of the network as a
whole. Other improvements include an update to HTTP2 process that the Hotspot uses to communicate to
Validators (over GRPC), and a Kerlink-specific improvement that fixes the RSSI computation if the
SNR is lower than zero to reduce the number of invalid Proof-of-Coverage challenges and receipts.
On Monday 2022-08-08 at approximately 19:34 PM UTC Validators stopped producing blocks at block
1,469,951 and a chain halt occurred. Normal chain operations resumed at approximately 08:00 AM UTC
just over 7 hours later.
MOBILE support is here! Once you have MOBILE in your account, you'll be able to see its balance and send MOBILE payments to other Wallets on the network.
The team has released the latest version of Console 2.2.20 to production. This release includes a
rollup of a number of updates:
The team is prioritizing its focus on issues to improve network reliability. Any new features or enhancement requests, while appreciated, will be added to our backlog and prioritized accordingly.
The core developers have tagged a new 2022.08.02.0 miner release. This release contains
improvements to packet delivery such that when a Hotspot sees a packet, it sends the payload right
away to the Router (and thus, recipient) immediately. Data transfer reliability is expected to
improve with this release.
The team has released the latest version of Console 2.2.17 to production. This release includes a
rollup of out of band releases:
The Nova developers have tagged a new 2022.07.15.0 firmware release for Original Helium Hotspots.
This minor release enables mux and disables blockchain sync. Enabling mux allows the transferring of
packets to use the Light Hotspot part of the codebase and is a further step forward in the Light
Hotspot transition.
Sprint: 80
The team has released the latest version of Console 2.2.13 to production.
The core developers have tagged a new 2022.07.14.0 miner release. This release contains fixes to
packet delivery to make it more reliable when Hotspots fall offline for a brief period of time (it
will queue up to 10 device packets it saw) and fixes to let Hotspots send unknown packets if it's
not synced with the blockchain.
On Monday 2022-07-11 at approximately 12:42 PM UTC the Consensus Group stopped producing blocks at
block 1,435,692. Normal chain operations continued at 17:15 PM UTC just over 4 hours later.
Sprint: 79
The team has released the latest version of Console 2.2.12 to production.
Users that have imported their Ledger Nano S (Android Only) or Ledger Nano X can now send payments from the app! Requires Ledger Helium App version 2.2.4. Update your Nano app using Ledger Live.
Sprint: 78
The team has released the latest version of Console 2.2.11 to production.
The core developers have tagged a new 2022.06.09.0 miner release. This release contains fixes to
the packet issue identified in the last release by bumping the gateway-rs version to alpha-27,
fixes a crash in ebus (more on this below), and a "flatline" fix (more on this below).
Sprint: 77
The team has released the latest version of Console 2.2.10 to production.
The core developers have tagged a new 2022.06.02.1 miner release. This is an emergency release
to address issues found in 2022.06.01.0. A new dependency was introduced related to telemetry work
that wasn't properly configured for the release process.
The core developers have tagged a new 2022.06.01.0 miner release. This release is mandatory
for all Hotspot Makers. Contained in this release are further improvements to the flatline issue
(fixes HTTP2) and enables mux. Enabling mux allows all data packet traffic to be routed through
gateway-rs and away from the erlang miner. This is a key step towards removing Erlang from Hotspots
altogether.
The core developers have tagged a new 2022.05.24.0 miner release. This release is mandatory
for all Hotspot Makers. Contained in this release are chain variables to support Validator
Performance (to be activated some time after May 25 9:00AM PT). Failure to update your fleet before
the chain variables are activated will cause the Hotspot to stop functioning as Hotspots still
follow the blockchain.
Sprint: 76
Despite all the excitement the team has released the latest version of Console 2.2.9 to
production.
The core developers have tagged a new 2022.05.19.0 miner release. This release is highly
recommended for all Hotspot Makers.
This update adds Light Hotspot support to the app, specifically for Transferring Hotspots and removes pre-Light Hotspot terminology from Hotspot details and Diagnostic Reports.
The core developers have tagged a new 2022.05.13.0 miner release. This release is highly
recommended for all Hotspot Makers.
This update is to fix a few bugs found since the last release. The next release will address changes for Light Hotspots.
The core developers have tagged a new 2022.05.12.0 miner release. This release is highly
recommended for all Hotspot Makers.
The core developers have tagged a new 2022.05.11.0 miner release. This release is highly
recommended for all Hotspot Makers.
The core developers have tagged a new 2022.05.10.0 miner release. This release is highly
recommended for all Hotspot Makers.
Sprint: 75
The core team has released the latest version of Console 2.2.8 to production. Changes include merging the code for XOR filter fee reduction, however it has not been activated as the vote is still ongoing. For more information check the HIP.
With HIP 55 the community approved increasing the scope of work and rewards for Validators. This document includes preliminary estimates to help provide further insight into the changes Light Hotspots have on Validator traffic.
The core developers have tagged a new 2022.04.27.0 miner release. This release is mandatory
for all Hotspot Makers.
Sprint: 74
The team has released the latest version of Console 2.2.7 to production. Changes include MQTT
connection overflow fixes, as well as usability and open source updates, for example payment
processor Stripe is available to open source users, and new environment variables have been added to
the .env template file.
The core developers have tagged a new 2022.04.19.0 miner release. This release is mandatory
for all Hotspot Makers. We request that Hotspot Manufacturers upgrade their fleets by 00:00 UTC on
2022-05-03. This update contains Light Hotspot software, and when activated via chain variable on
May 3 2022, will change how Hotspots receive challenges, remove block sync from Hotspots, and
stabilize Hotspot performance. This will allow the approved
HIP 55 chain variable to be
activated and prepare for the activation of Light Hotspots.
The Nova developers have tagged a new 2022.04.14.0 firmware release for Original Helium Hotspots.
This minor release increases the interval a Hotspot checks for a new miner container from 4 hours
(6 times a day) to 24 hours (once per day).
Sprint: 73
The core team has released Console v2.2.6 to production. A number of updates in this sprint
including increasing number of packets for multi-buy, clearer messaging for unsuccessful
integrations, and a pre-built Integration with Microsoft Azure IoT Central, instructions here:
https://docs.helium.com/use-the-network/console/integrations/azure/
The Nova developers have tagged a new 2022.04.05.0 firmware release for Original Helium and
RAK/MNTD Hotspots.
Requires users to enter their 12 word seed phrase in order to sign out to prevent accidental sign outs. Thanks to our community Mentor for writing this request!
iPhone users - our community member Perroni_F5 has added an incredible feature to the Hotspot app, adding 2 widgets to your app experience:
The Nova devs have tagged a new 2022.03.31.0 firmware release for Original Helium and RAK/MNTD
Hotspots.
The Helium team has tagged a new 2022.03.29.1 firmware release for Original Helium and RAK/MNTD
Hotspots.
The core developers have tagged a new 2022.03.23.1 miner release. This release is mandatory
for all Hotspot Makers. We request that Hotspot Manufacturers upgrade their fleets by 12:00 UTC on
2022-03-29. This will allow the approved
HIP 54 chain variable to be
activated and ensure Hotspots can continue creating Challenges using the new H3Dex-based PoC
targeting.
The core developers are changing the PoC Interval from 475 to 950. Hotspots will PoC less often
however total average rewards will remain unchanged. PoC rate has no effect on average individual
Hotspot earnings, but may increase variability of rewards. This will help the chain process
transactions more efficiently. Please read the details below for our reasoning.
Sprint: 72
Console v2.2.5 has been released to production. During this sprint the team also launched Helium
Roaming Services. LoRaWAN network providers can learn more about how to leverage Helium coverage
here.
The Helium team has tagged a new 2022.03.15.0 firmware release for Original Helium and RAK/MNTD
Hotspots.
Sprint: 71
The team has released v2.2.4 release to production. A number of usability updates were completed
including minimizing user input for Azure IoT Hub, making it simpler to add a label to new devices,
and adding abiliity to use mobile UI to scan QR codes for devices that have implemented the LoRaWAN
Device Identification QR Codes specification.
The core developers have tagged a new 2022.03.07.0 miner and blockchain-core release. This
release is recommended for all Hotspot Makers.
When browsing the network map on the app, Hotspots on the denylist will appear with a red badge next to their name. Denylist Hotspots are those identified to be gaming on the Helium Network.
The core developers have tagged a new 2022.02.28.0 miner and blockchain-core release. This
release is recommended for all Hotspot Makers.
Sprint: 70
The latest v2.2.3 release has been releaseed to production by the team. Based on a number of conversations with users, a large number of Console API’s were added to this release and existing ones updated.
The core developers have tagged a new 2022.02.22.0 miner and blockchain-core release. This
release is mandatory for all Hotspot Makers (please update your fleet by March 3, 2021 00:00
UTC) and will be mandatory in the near future for Validators, ETLs, Routers and Nodes. It contains a
network improvement that is gated behind some currently inactive chain variables. Hotspots without
this update will not be able to sync or mine once the chain variable is activated later.
The Helium team has tagged a new 2022.02.17.0 firmware release for Original Helium and RAK/MNTD
Hotspots.
On Monday 2022-02-07 5:46 AM UTC the team announced a chain halt caused by enlarged block 121528. Normal chain operations were announced at 3:04 PM UTC, less than 10 hours later.
Sprint: 69
The team has pushed the latest v2.2.2 release to production. Highlights include updates to Receive Window Delay feature, added message queue (for Production Console only) to scale frontend database, and ux tweaks. In addition, for open source users the front end team added prebuilt Console images hosted on Quay.io so open source users do not have to build their own Console images on docker.
The Helium team has tagged a new 2022.02.10.0 firmware release for Original Helium and RAK/MNTD
Hotspots. All Original Helium/RAK MNTD Hotspots have a designated hardware_type to identify its
hardware version at the manufacturing stage. By targeting Hotspots by its hardware_type, this
release will enable RAK/MNTD Support teams to investigate and diagnose Hotspot-related issues in a
secure manner without interaction from the Helium Support team.
A new way to transfer Hotspots is now available on the app. Now transferring Hotspots is as easy as sending a payment transaction. Hotspot sellers can initiate the transfer from Hotspot Settings, the same as before, but instead of waiting for the buyer to accept the transaction, the transfer is immediately submitted to the blockchain. Once cleared, the buyer will see the new Hotspot in their app. When the transfer is successful, both buyers and sellers will receive a notification.
The Helium team has tagged a new 2022.02.08.0 firmware release for Original Helium and RAK/MNTD
Hotspots. This minor release changes the interval between automated update checks for a Hotspot's
miner container. It has been updated to every 2 hours, from the current 5 minute interval. This
update also changes how a Hotspot prunes old miner images, removing the possibility of
re-downloading the latest image multiple times. These changes should reduce the amount of bandwidth
an Original or RAK/MNTD Hotspot consumes.
Note: This release is only relevant to RAK/MNTD and Original Helium Hotspots.
Sprint: out of band
This is an out of band release that addresses the latest issue related to receive window delay, specifically this issue.
The core developers have tagged a new 2022.01.29.0 miner release. This release is recommended
for all makers, but is only mandatory for Hotspots that are no longer able to load the latest
snapshot. As the network gets larger and the snapshot size increases, Hotspots with less RAM will
have trouble loading them. We will continue to support these Hotspots until all manufacturers have
an upgrade path to Light Hotspot firmware.
The core developers have tagged a new 2022.01.27.0 miner release. This release is recommended,
but not mandatory, for all Hotspot manufacturers.
Sprint: 68
This release includes mostly backend under the hood fixes. The team plans to do an out of band release next week to address the latest issue related to receive window delay, specifically this issue.
Witnessed Lists in both the App and Explorer reflects the list of Hotspots that your Hotspot has seen beacon. This is an improved change from displaying a list of Hotspots that has witnessed your Hotspot as the new list directly correlates with mining rewards.
This release decouples miner releases from Hotspot firmware updates by running miner in a self-updating container.
Sprint: 67
Based on almost 5000 responses to our Console UI survey, 70% respondents reported accessing Console at least multiple times a week, and almost 50% daily.
The core developers have tagged a new 2022.01.12.1 Hotspot firmware release which includes
recommended improvements to snapshot loading and reduces their memory footprint. Furthermore,
this release also contains bug fixes for state channel handling specifically for the client side
which should now clean the internally stored channels appropriately when they observe a
state_channel_close transaction.
The core developers have tagged a new 2022.01.04.0 Hotspot firmware release which includes
recommended improvements in miner specifically related to ensure that Hotspots with unasserted
location are still able to sync to the head of the blockchain. Secondly this release also includes
ability to use additional packet metadata provided by certain packet forwarders.
The core developers have tagged a new 2021.12.29.2 Hotspot firmware release which includes
recommended improvements in blockchain-core specifically in how Snapshots are loaded and
blocks are stored locally on Hotspot disks.
See how your Hotspot stacks up to the Network! We’ve added the average expected network earnings per Hotspot to see how yours perform against other Hotspots on the network. The average earnings is based on how many Hotspots are online that day, for creating challenges, beaconing, witnessing, and transferring data.
Sprint: 66.5
The latest version of Console (sprint 66.5) has been deployed to production.
Sprint: 66
The latest version of Console (sprint 66) has been deployed to production. Due to holidays we had an extended sprint 65, which shortened sprint 66. T The team also began sprint 67 which will be the last sprint of 2021 with a release planned for early January 2022.
The Helium Hotspot firmware developers have identified an issue with how the LoRaWAN packet forwarder is configured on sx1302-based Helium and RAK Hotspots after PoCv11 activation. We are issuing an emergency release to these devices to address this issue.
The core developers have tagged a new 2021.12.14.0 Hotspot firmware release which includes
recommended bug fixes in erlang-libp2p and blockchain-core specifically in Snapshot loading
and UPnP.
Sprint: 65
The latest version of Console (sprint 65) was released to production, including user enhancements, multi-factor authentication, and a host of backend updates.
The core developers have tagged a new 2021.12.09.0 Hotspot firmware release which includes a
recommended bug fix in erlang-libp2p. This change potentially mitigates the zero-witness
beacon issue that has been afflicting, mostly, relayed Challenger Hotspots failing to receive
witness receipts. Special thanks to @groot#3396 and others in the #poc-discussion channel on Discord
for their efforts in tracking this issue down.
Several users have noted that submitting the antenna and elevation was not working as expected.
The core developers have tagged a new 2021.12.06.0 Hotspot firmware release which includes
recommended fixes to Seed nodes and DNS query improvements.
Updates Hotspot Earnings chart to show UTC days (7D, 14D, 30D).
Last week the Helium Network experienced several disruptions. Throughout the disruptions, there was no risk to HNT holders, and while some devices were unable to use the Network for data transfer, the majority of devices were unaffected.
The core developers have tagged a new 2021.11.30.1 Hotspot firmware release which includes
recommended fixes to snapshot load performance, underlying rocksdb tuning and other important
bug fixes.
The core developers have tagged a new 2021.11.22.0 Hotspot firmware release which includes a
mandatory fix to unstick some Hotspot stuck syncing at 1,107,994 due to a nonce disagreement
from loading a snapshot.
The core developers have tagged a new 2021.11.21.2 Hotspot firmware release which includes
highly recommended fixes to improve network-wide block sync and snapshot ingestion on Hotspots
with less memory.
The core developers have tagged a new 2021.11.19.0 Hotspot firmware release which includes
highly recommended fixes to improve network-wide block sync and snapshot ingestion on Hotspots.
This release includes improvements to API loading, adds rewards caching, and updates how earnings are calculated.
Sprint: 64
Despite additional turbulence experienced earlier this week, the team stayed focused and released v2.1.6.
The core developers have tagged a new 2021.11.17.2 Hotspot firmware release which includes
CRITICAL fixes to ensure that Hotspots continue to follow the chain after the
24 hr+ chain halt observed on November 15th.
The core developers have tagged a new 2021.11.11.0 Hotspot firmware release which includes
critical bug fixes for gossip and ARP in libp2p. We hope that this release will help Hotspots with
improvements on block gossip and keeping an up to date peerbook.
The core developers have tagged a new 2021.11.10.0 Hotspot firmware release which includes
critical bug fixes, supplementary code for finalizing activation of POC V11 related chain variables,
packet transmission and blockchain-etl fixes.
The core developers have tagged a new 2021.11.04.2 Hotspot firmware release which includes a few
minor usability improvements, specifically allowing Makers to access a Hotspot's region parameters
via JSON-RPC and miner's CLI.
The team added another prebuilt integration with a partner akenza IoT platform, as well as updates to Configuration Profiles, and the Coverage feature.
The core developers have tagged a new 2021.11.02.0 Hotspot firmware release which includes two
critical blockchain-core fixes, firstly, blockchain core can now handle v1 and v2 block gossip
seamlessly; secondly, the developers noticed that the transaction_validity_version chain variable
definition was set incorrectly in the header file, this has been fixed. An unfortunate effect of
this is that transfer_hotspot_v2 support has been slightly delayed.
On 10/27 a fix based on a community-raised issue was deployed along with a number of other updates
in the 2021.10.27.0 Hotspot firmware release.
The core developers have tagged a new 2021.10.29.0 Hotspot firmware release which includes bug
fixes for dropped packets reported by the community and verified by the router team.
The core developers have tagged a new 2021.10.27.0 Hotspot firmware release which includes new
functionality for Hotspots to automatically ingest latest on-chain snapshots. Makers, please see
the note below as there are some updates you'll need to incorporate to use this feature. This
release also reverts the var cache which was causing unforeseen issues and upgrades
blockchain-core to include many State Channel related optimizations.
App Version 3.7.0
We've added StakeJoy API support to the Helium App. StakeJoy provides data to explorer-old.helium.com and now the Helium app to provide users with an improved experience. To use StakeJoy, go to Settings, tap Choose API Provider and select StakeJoy.
Included in this sprint, the team delivered Configuration Profiles provides users with a streamlined way to configure, save, and apply device settings (ADR, CF List) and a new authentication system.
The core developers have tagged a new 2021.10.18.0 Hotspot firmware release which includes some
bug fixes and a new blessed snapshot at height 1059841. This release addresses a channel selection
bug causing unnecessary incorrect witnesses and fixes a potential race condition when initializing
var-cache. Furthermore, the updated blockchain-core has improved state channel support for
router.
This post-mortem describes the events leading up to the app outage on October 17, what happened, and what the team is doing moving forward.
The core developers have tagged a new 2021.10.07.0 Hotspot firmware release which includes several
updates.
State channel updates to improve network performance was released to resolve a choke point caused by the exponential growth of Hotspots providing coverage. In addition, Flows has become popular to quickly send device data over The People’s Network and this release includes a number of usability updates.
App Version 3.6.0
Ever need to hand your phone to someone for installation but don’t want them seeing your seed phrase, use the wallet payment functionality, or transfer Hotspots?
The core developers have tagged a new 2021.09.27.0 Hotspot firmware release which includes some
performance fixes and a bug fix to block sync. It also includes a new core developer blessed
snapshot at height 1029491. We've identified the cause of lack of agreements on on-chain snapshots
and have issued a fix for it as well but this will require a validator release.
The core developers have tagged a new 2021.09.26.0 Hotspot firmware release which primarily
includes networking, PoCv11 and transaction manager bug fixes. It also includes a new core developer
blessed snapshot at height 1024971 while we wait for the validator pool to update to version
1.3.3.
This release includes an integration for Microsoft Azure IoT Hub. Microsoft had requested an integration with Console based on feedback from their customers. In addition, the team made usability updates based on community feedback, and enhanced device APIs.
The core developers have tagged a new 2021.09.16.1 Hotspot firmware release which primarily
includes performance improvements for Hotspots. You can read about them below. This release also
updates the blessed snapshot to 1012321.
The core developers have tagged a new 2021.09.14.0 Hotspot firmware release which contains several
updates including support for PoCv11 (not yet activated), updates to snapshot loading, several
libp2p and rewards performance changes.
A major feature the team’s worked on for the last two sprints is Coverage. Unlike Hotspot monitoring tools focused on maximizing mining rewards, Coverage is built for users of the network who want to have more visibility about which Hotspots are transferring device packets as well as displaying the status of those Hotspots all from within Console.
App Version 3.5.0
Have a Hotspot that has met their untimely demise and the daily reminders just reminds you of the good old days and it just hurts too much? Or maybe you want to hide it for other reasons - we don't judge. Now you can hide it from app view (but it’ll stay associated with your account). To unhide Hotspots, go to Settings to view hidden Hotspots.
The core developers have tagged a new 2021.09.03.0 Hotspot firmware release which contains a
snapshot update. Please note that this tag is based off of 2021.08.31.1 and not on miner master as
the team wants to continue testing PoCv11 changes that landed on master yesterday.
The core developers have tagged a new 2021.08.31.1 Hotspot firmware release which contains major
blockchain and libp2p stability and performance updates. This is a mandatory release for all
hotspot manufacturers to continue syncing the blockchain.
While the team focused on the Gateway Monitoring feature, this sprint they also knocked off a number of frontend and backend issues reported by the community. In addition, an out-of-band update was released last week to address a number of issues.
The core developers have tagged a new 2021.08.24.4 emergency Hotspot firmware release which
contains important snapshot load fixes and a new blessed snapshot. This is a mandatory release for
all hotspot manufacturers to enable the loading of new snapshots.
The core developers have tagged a new 2021.08.21.0 Hotspot firmware release which contains major
blockchain-core updates specifically ensuring that snapshots are correctly getting loaded.
Furthermore this release supplies a new snapshot to allow new and un-synced hotspots to continue
syncing. This is mandatory release for all hotspot manufacturers.
The core developers have tagged a new 2021.08.16.0 Hotspot firmware release which is a maintenance
update to primarily update static seed nodes in miner. This is an optional but recommended release
for Hotspot owners and manufacturers to improve network performance.
Support for Data-Only Hotspots
The core developers have tagged a new 2021.08.11.0 Hotspot firmware release which allows a new way
to fetch seed nodes from DNS, updates the currently blessed snapshot, and contains a few other small
fixes
The core developers have tagged a new 2021.08.05.0 Hotspot firmware release which updates the
currently blessed snapshot and includes some fixes for state channel handling for packet routing.
The release also includes support for fetching more seed nodes from DNS and some fixes for other
applications using blockchain-core (router and ETL).
The core developers have tagged a new 2021.08.02.0 Hotspot firmware release which updates the
currently blessed snapshot and handles IPv6 addresses in bogon check. This release also removes the
default Helium production router from the static configuration to help meet the demands on
utilization on the network
The core developers have tagged a new 2021.07.30.0 Hotspot firmware release which updates the
currently blessed snapshot, fixes a minor bug in snapshot downloads from the Amazon S3 source, and a
community contribution from @PaulVMo that improves libp2p
identification of private IP addresses. This is an optional but recommended release for Hotspot
owners and manufacturers to continue syncing with newly blessed snapshots.
The core developers have tagged a new 2021.07.29.0 Hotspot firmware release which updates the
currently blessed snapshot, adds a new seed node to the statically configured list, adjusts address
hashing for Hotspot identification, and contains minor bug fixes. This is an optional but
recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed
snapshots.
After many months of work and testing based on community feedback, the team is excited to announce the official release of Console 2.0! The team not only worked on new features and capabilities, but also focused on making sure migration to Console 2.0 would be seamless for existing users.
The core developers have tagged a new 2021.07.22.0 Hotspot firmware release which updates the
currently blessed snapshot and contains networking and snapshot improvements. This is an optional
but recommended release for Hotspot owners and manufacturers to continue syncing with newly
blessed snapshots.
The core developers have tagged a new 2021.07.19.0 Hotspot firmware release which updates the
currently blessed snapshot, introduces state channel related improvements and switches all Hotspots
to non-validation (follow mode). This is an optional but recommended release for Hotspot owners
and manufacturers to continue syncing with newly blessed snapshots.
After many months of work and testing based on community feedback, the team plans to release Console 2.0 on Thursday 7/29.
This release the team worked on addressing issues reported for Console 2.0 beta and also backend fixes to make joins and packet transfer more reliable.
The core developers have tagged a new 2021.07.15.0 Hotspot firmware release which introduces more
snapshot related fixes. This is a mandatory release for Hotspot owners and manufacturers to
continue syncing with newly blessed snapshots.
The core developers have tagged a new 2021.07.13.1 Hotspot firmware release which contains
snapshot ingestion improvements and a newly blessed snapshot. This is a mandatory release for
Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.
The core developers have tagged a new 2021.07.11.0 Hotspot firmware release which contains
performance improvements to snapshot ingestion. This is an optional but highly recommended
release for Hotspot owners and manufacturers especially for those Hotspots which are based on older
Raspberry Pi3 hardware.
The core developers have tagged a new 2021.07.09.2 Hotspot firmware release which contains a new
blessed snapshot and gossip improvements. This is an optional but recommended release for Hotspot
owners.
The core developers have tagged a new 2021.07.02.0 Hotspot firmware release which contains a new
blessed snapshot and minor maintenance updates. This is an optional but recommended release for
Hotspot owners.
This week we’re excited to release Console 2.0 beta on our staging server. Console 2.0 beta represents months of work by the team and we’re excited to share with our community and get their feedback.
The core developers have tagged a new 2021.06.30.0 Hotspot firmware release which contains updates
to improve peerbook performance and a new blessed snapshot. This is an optional but recommended
release for Hotspot owners.
• Show Hotspot's gain and elevation
• Adds initial transaction support for staking transactions
The core developers have tagged a new 2021.06.29.0 Hotspot firmware release which primarily
contains maintenance updates to potentially reduce block times and enhance block sync. This is an
optional but recommended release for Hotspot owners.
The core developers prepared an emergency release 2021.06.25.3 to fix the Mainnet blockchain halt
noticed around 4:00 PM PST, June 25th, 2021. This was a mandatory release for all blockchain
participants on the network. Hotspot manufacturers who have not picked up this release should update
their fleets. Please read further for more details.
The core developers have tagged a new 2021.06.25.0 Hotspot firmware release which contains updates
to support better transaction validation for Validators and improvements to State Channel
scalability. This is a required release for Hotspot owners as it introduces new chain variables
which will get activated at a later date.
The core developers have tagged a new 2021.06.21.0 Hotspot firmware release which only has a new
blessed snapshot update. This is an optional but recommended release for Hotspot owners; the updated
blessed snapshot will help new and lagging Hotspots to sync to blockchain head faster.
This release fixes 2 bugs:
The core developers have tagged a new 2021.06.18.0 Hotspot firmware release which contains network
improvements related to NAT and bug fixes related to staking for Validators. This is a required
release for Hotspot owners. As some of the community members have noticed that the blockchain has
not generated a new snapshot since the last GA, we believe that the improvements to networking and
an increment to snapshot memory generation limit will fix snapshot generation. The core developers
plan to do another minor release with a snapshot update once it gets generated by the blockchain.
The team focused on delivering an pre-built integration for Microshare in addition to continuing to build out our roaming capabilities for non-Helium traffic.
Hexagons on the Helium Network
The core developers have tagged a new 2021.06.14.0 Hotspot firmware release which contains
transaction processing improvements, preparatory code for a new chain variable, and a newly
generated snapshot. This is a required release for Hotspot owners. Further announcements will be
made for chain variable activation schedule but we expect about a week or so of time to allow
downstream clients of the chain (etl, node, and router users) to update.
The core developers have tagged a new 2021.06.11.2 Hotspot firmware release which has minor test
related fixes and a newly generated snapshot. This is an optional release for Hotspot owners but
recommended to get newly onboarded and/or un-synced hotspots to sync to the head of the blockchain.
The core developers have tagged a new 2021.06.09.0 Hotspot firmware release which contains a newly
generated snapshot. This would allow newly onboarded and previously un-synced hotspots to sync to
the head of the blockchain much faster.
The core developers have tagged a new 2021.06.08.0 Hotspot firmware release which fixes Snapshot
creation and ingestion. It also adds support for storing transaction with implicit burns. This is a
required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware
when we release to General Availability.
This sprint we included updates for downlinks including refresh and showing payloads in both base64 and decoded text.
Bug Fixes
As some of the community has noticed, block times and elections have been struggling the last week
or so as the network continues to grow. The core developers have prepared the 2020.05.29.1 release
which will adjust validation times for two known slower transactions in our blockchain. This is an
optional but recommended release for miners on the network.
The core developers have prepared an emergency release 2021.05.29.0 to fix the Mainnet chain halt
noticed around 1:00 PM PST, May 29th, 2021. This is a mandatory release for all miners on the
network. Please read further for more details.
The core developers have tagged a new 2021.05.28.0 Hotspot firmware release which prepares support
for Validators on Mainnet. This is a
required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware
when we GA.
The core developers have tagged a new 2021.05.24.1 Hotspot firmware release which changes S3
snapshot download timeout to a more relaxed limit and updates the blessed snapshot to 858241. This
is an optional, but recommended upgrade for all miners and will be automatic for all Hotspots using
the Helium firmware when we GA.
The core developers have tagged a new 2021.05.22.0 Hotspot firmware release which adds S3 download
functionality for snapshots and updates the blessed snapshot to 855361. This is an optional, but
recommended upgrade for all miners and will be automatic for all Hotspots using the Helium firmware
when we GA.
The team delivered the ability to view LoRaWAN MAC command information for downlinks. Also along with other updates backend work was done to enable Discovery Mode for Hotspots. See details about Discovery Mode here.
The core developers have tagged a new 2021.05.20.0 Hotspot firmware release which primarily fixes
libp2p peerbook issues affecting a small number of Hotspots and also updates the blessed snapshot.
This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium
firmware when we GA.
The team has a new release tagged 2021.05.19.0 which primarily fixes low space observed on some
Hotspots with smaller SD cards and updates the blessed snapshot. This is a required upgrade for
all miners and will be automatic for all Hotspots using the Helium firmware when we GA.
The team has a new release tagged 2021.05.18.0 which primarily fixes out-of-memory and sync issues
observed on older, less capable hardware, and other bug fixes. This is a required upgrade for all
miners and will be automatic for all Hotspots using the Helium firmware when we GA.
Use Discovery Mode to understand your Hotspot’s coverage without waiting for Proof-of-Coverage/Beacons, or setting a location!
The Helium core developers have prepared a new blockchain release tagged 2021.05.16.1 which makes
a number of changes to help older Hotspots be able to sync the blockchain. This is a required
upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.
The team has a new release tagged 2021.05.12.0 which adds support for light gateways and miner
discovery mode. This is a required upgrade for all miners and will be automatic for all Hotspots
using the Helium firmware when we GA.
Hotspots that are relayed through other Hotspots on the network may experience syncing or reduced mining rewards. We now surface that up to the user in the app (already exists on explorer-old.helium.com).
Last night, during the rollout of the 2021.05.04.2 release including the code for
HIP-29 and
HIP-30, several
members of the community observed a long election including multiple proposed Consensus Groups where
only 15 of the 16 new members succeeded in their Distributed Key Generation (DKG). After several
hours of monitoring the Consensus Group, the Decentralized Wireless Alliance (DeWi) in conjunction
with several core developers ended the 450 block epoch with a Rescue Block, which later restored the
network to a functioning state. The network has been operating correctly since. All funds remained
safe through the entire process.
This sprint the team focused on Integration updates including building a Google Sheets prebuilt Integration, adding dynamic query parameters, and including a hold_time field to increase accuracy of timestamps.
A modified Hotspot UI that allows for navigating to the next Hotspot in a list, without going back to the Hotspot tab.
The team has a new release tagged 2021.04.26.1 which contains some new features and bug fixes.
This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium
firmware when we GA. Specifically, this release includes support for Payment Memos which requires a
chain variable to activate and makes this a required release. We don't intend to activate this chain
variable for about a week. Please read further for more details.
This release was the first with our three-tier development environment and focused on improved downlink capabilities and provided more details for LoRaWAN MAC commands for uplink events.
Remote Assert is finally here! Update your Hotspot's location without being physically next to the Hotspot, and without Bluetooth pairing! This improved user experience will make it much easier for Owners and Hosts to set up Hotspots.
The team has a new release tagged 2021.04.14.0 containing an OTA Server Certificate Update.
Here is the latest update from our Makers manufacturing Hotspots for the Helium Network.
The team has a new release tagged 2021.04.12.0 which contains a some fixes that should allow older
Hotspots to keep consuming the chain, as well as a snapshot bump.
The team has a new minor, but urgent, release tagged 2021.04.09.0 which contains a bug fix to
libp2p. A latent bug was uncovered when we fixed another issue with the prior release. We are also
updating the snapshot with this release.
The team has a new minor release tagged 2021.04.08.0 which contains a minor bug fix that addresses
an issue that prevented snapshots from being generated by the chain. We are deploying this change
today so that we can do a subsequent bump of the snapshot tomorrow. A list of other minor changes
are below.
Starting on April 4, our daily active users (DAU) on Helium Explorer doubled from 15,000 to 30,000. As a result, the API services that Explorer relies on was unable to handle the additional load. The reason for this cannot be fully attributed to DAU, but rather a symptom of the root cause. This API is used by multiple services including the Explorer, Helium App, and Wallet CLI. Many third party services also depend on this API.
The team has a new minor release tagged 2021.04.05.0 which contains a minor bug fix that caused
the miner to restart every few hours, plus a snapshot bump.
The team has a new release tagged 2021.04.02.0 which contains a regular maintenance update with a
support for assert_location_v2 and a snapshot bump.
The team made a number of core backend fixes related to downlinks, and initial device joins. Debug mode, and the event log also received updates.
The team has a new minor release tagged 2021.03.26.0 which contains a regular maintenance update
with an improvement to the performance of distributed key generation used in consensus and a
snapshot bump.
The team has a new release tagged 2021.03.24.0 which contains a significant overhaul of the
Rewards transaction and maintenance improvements including a regular snapshot bump.
Show which Maker paid for the Add Hotspot and Assert Location transaction
The team has a new minor release tagged 2021.03.22.0 which contains a regular maintenance update
with an update to the countries supported by the miner codebase and a snapshot bump.
Since the last update in February, Makers Nebra, Syncrob.it and Bobcat have concluded their Audit.
G’day mates. In addition to support for Australia 🇦🇺 (AU915), significant work to improve both the accuracy and scalability of events was completed. We also added a pre-built integration for the Tago.IO IoT platform.
The team has a new minor release tagged 2021.03.18.0 which contains a regular maintenance update
with a snapshot bump and a minor update to the address hash.
We have discovered an issue, this week, that affects a number of RAK Hotspot Miners shipped by Cal-Chip to customers starting January 2021 until today (the "Dolphin" batch, up to order #2999). This issue is the same as November’s discovery.
New Features:
The team has a new release tagged 2021.03.08.0. This release includes a fix to Hotspot firmware
where onboarding keys were incorrectly generated, a snapshot bump to 750241, and several
reliability fixes that were identified in our Validators Testnet.
We’re excited to announce the release of Helium Console under the Apache 2.0 license. Console joins a growing list of Helium open source projects including blockchain core, the wallet, the blockchain explorer, the Helium app and others
The team has a new release tagged 2021.03.05.0. This release collects some small but important
maintenance upgrades to the blockchain but primarily is focused on adding a new functionality to the
Proof of Coverage system. We also included a few code changes that will support Light Hotspots in
the future and snapshot bump.
The team has a new release tagged 2021.03.02.2. This release collects several performance and
maintenance upgrades to the blockchain that should improve stability and performance of members of
the Consensus Group. We also included a few code changes that will support Light Hotspots in the
future, a bugfix for oracle prices our testnet chain, and snapshot bump.
New Features:
Show the time period when scrubbing charts on Hotspot Details
Make addresses compatible on the Network selectable: either copy to clipboard or open in explorer
The team has a new release tagged 2021.02.26.0. This is mostly minor from a Hotspot owner
perspective but there are some major infrastructural improvements which will help us continue to
maintain the code base including an upgrade to Erlang/OTP 23. We also included some region updates
and a snapshot bump.
After months of waiting, we’re so happy to announce the new version of the Helium Hotspot app is here! Version 3.0 introduces a whole new design and introduces a new way to interact with the app. Gone are the days of scrolling for hours toiling over challenges (we have Explorer for that!).
The team has a new minor release tagged 2021.02.23.0 which contains a regular maintenance update
with an update to the seed nodes and a snapshot bump.
The team has a new minor release tagged 2021.02.18.0 which contains a regular maintenance update
with a snapshot bump.
This release the team added another pre-built integration with Ubidots and extended regional support for China. In addition, the team completed a number of usability enhancements, including making it easier to add Labels to devices, and identify which devices have sent data.
The three newly approved Makers of the Helium network have made great progress in their audit and integration with the Hotspot app. We’d like to take this opportunity to inform the community on their progress and how long each Makers took to progress through the milestones so you, as a potential Hotspot owner, can understand when to expect your Hotspot once a Maker starts their audit process.
The team has a new minor release tagged 2021.02.12.0 which reverts the RSSIS change from
2021.02.10.4. This change, while correct, interacts poorly with our existing PoC acceptance
criteria and we're reverting it temporarily while we fix those.
The team has a new minor release tagged 2021.02.10.4 which contains regular maintenance updates.
Please read further for more details.
The team has a new minor release tagged 2021.02.09.2 which contains a reliabiliy fix for consensus
group members and some CLI fixes.
The team has a new minor release tagged 2021.02.05.0 which contains an important NAT fix and a
regular snapshot bump. Please read further for details.
We’re excited to launch a beta release of Flows, a new user experience designed to simplify performing Console core actions. Also, in this release notifications can be sent via webhooks, and APIs for notifications have been added.
The team has a new release tagged 2021.02.01.1 which contains minor bug fixes, routing transaction
updates and a snapshot bump. This is a required upgrade for all miners and will be automatic for
all Hotspots using the Helium firmware when we GA. Please read further for more details.
The team has a new release tagged 2021.01.27.0 containing a change to the Radio Power Output,
updates to RSSI offset, support for the onboarding server, bug fixes, and a snapshot bump.
As outlined in the blog posted January 20, 2021, the team is making a change to how Hotspots are onboarded to accommodate our new third-party Hotspots and manufacturers.
The Helium community places a lot of trust in HIP19 approved hardware vendors and grants them the ability to add full mining gateways to the Helium blockchain. These manufacturers, also known as “Makers”, are limited by chain variables and the Onboarding Server as described previously on the Helium blog. The community trusts that the Maker will produce and deliver a real physical Hotspot with approved security methods.
In this release we’ve included beta support for LoRaWAN adaptive data rate (ADR), increased the number of email notifications, and added a new API for multiple packet purchase. ADR allows devices to use an optimal data rate which reduces power consumption and airtime on the network based on RF conditions. It's recommended to only use this setting for fixed or non-mobile devices to ensure reliable connectivity.
The team has a new emergency release tagged 2021.01.17.0 which contains a bug fix to detect new
RAK Hotspot Hardware.
The team has a new release tagged 2021.01.12.0 which contains minor bug fixes, test updates and a
snapshot bump. Please read further for more details.
During the last sprint of 2020, the team delivered email notifications, the ability to pay for additional packets, and a number of backend updates. Configuration for email notifications and the ability to pay for additional packets has been added to Labels. This allows for flexibility over which devices trigger the notifications or receive additional packets.
The team has a new release tagged 2020.12.30.0 with improved hardware support for the RAK Hotspot
Miner updates (ECC chip support), new region configs for future hotspots in China, Japan, Malaysia,
Singapore and Australia, minor adjustments to challenge targeting jitter, and a snapshot bump.
Please read on for more details and plan for GA.
The Helium team will be adjusting the poc_v4_target_challenge_age chain variable from 300 to
1000 in order to align more closely with the current value of poc_challenge_interval (which is
currently set to 240).
The team has a new release tagged 2020.12.22.1 before the holidays with a snapshot bump and
blockchain core support for showing invalid witnesses in the API. Please read on for more details
and plan for GA.
The 2020.12.16.0 release is a minor blockchain maintenance release. The primary goal of this
release is to bump the blessed snapshot.
The team has successfully deployed support for HIP 15 and HIP 17 and switched to a beacon-only PoC model. One of the things we decided not to change with all of the other chain variables this morning was an increase to the number of max witnesses for each PoC. Now that we are confident with the way the Consensus Group is able to handle the new PoC receipts and new rewards calculations, we feel like it's safe to increase the max number of witnesses in each PoC receipt.
As approved by HIP13, the engineering team has finished the implementation of Transfer Hotspot, allowing any Hotspot owner to transfer ownership of a Hotspot to a new owner. The full help documentation can be found here: https://docs.helium.com/wallets/app-wallet/transfer-hotspot/ and CLI support here: https://developer.helium.com/blockchain/blockchain-cli#hotspots-transfer
The 2020.12.15.1 release is a bugfix release to add support for
HIP17.
Please find a detailed breakdown of the bugfixes below.
During this sprint we released Console backend components on our public github repo, made updates to JSON Messaging template, and made interface enhancements to help educate new users.
With the release of
HIP 16 and
implementation of
Random Consensus Group selection,
the PoC-based Hotspot Score is no longer a metric that's used by the Helium blockchain. We intend to
completely remove the score field from the
API response.
The Helium blockchain team will be submitting a chain variable transaction that will increase the
target interval between Proof-of-Coverage challenges (PoCs), changing the poc_challenge_interval
variable from 120 to 240 blocks. This update will effectively halve the rate of PoC transactions
that Hotspots generate.
This release was focused on Integration work, including adding a pre-built Adafruit IO Integration, and frontend updates to make the selected Integration more obvious.
The 2020.11.30.0 release is a minor blockchain maintenance release. The primary goal of this
release is to bump the blessed snapshot.
The team has been busy working on a new blockchain chain release tagged 2020.11.17.0. This release
contains a few new features:
This release includes a new Manager user role, and other usability updates as well the Cargo tracking application was released as open source.
The team has prepared a new blockchain chain release tagged 2020.11.10.0 addressing a key issue
where RAK Hotspot Miners would not properly update to new firmware OTAs.
In this new update, we've made it easier for anyone to send the full amount of HNT in their wallet. You no longer have to worry about calculating the transaction fee.
The team has noticed issues where a small number of RAK Hotspot Miners are not applying firmware updates containing bug fixes.
The team has prepared a second emergency weekend release, tagged 2020.10.31.1. This release
addresses a software issue that affects only a subset of the RAK Hotspot Miners, which were unable
to witness challenges or build witness lists.
The team has prepared an emergency weekend release, tagged 2020.10.31.0. This release addresses a
software issue that affects only the RAK Hotspot Miner where they were unable to witness
challenges or build witness lists.
In this release we provide support for a second receive window, a number of UI improvements, as well as a beta feature that allows users customize JSON messages sent to Integrations.
The 2020.10.27.0 blockchain release contains some critical fixes specifically meant to address the
blockchain halt observed on Monday, October 26th, 2020. Please read on for a detailed breakdown.
Update: The current release candidate primed for GA is 2020.10.23.0 which fixes minor state
channel server side issues. We will keep this post updated with more details.
In this new update, we've brought in a feature called "View Witnesses". This new feature can be found on your Hotspot's page. Simply toggle the View Witness to see all the Hotspots that your Hotspot has seen and interacted with!
This release includes enhancements to help manage large numbers of devices including ability to sort and customize the device table, and making group import more flexible. In addition, a number of backend fixes were made including improving join request reliability, making integrations more robust, and resolving an arbitrage issue.
The primary objective of the 2020.10.12.1 release is to address the missing data credit rewards
for blockchain_txn_state_close_v1 transaction. This release also fixes inconsistent JoinAccept
for packet transfers.
The primary objective of the 2020.10.08.0 release is to address the state channel overpayment
exploit. The fix is guarded behind reward_version chain variable which would be incremented to 4
to activate the fix. We intend to activate this chain variable after verifying this release.
In order to comply with local North American and European regulations, we added an undocumented update to PoC to correct the spreading factor used by PoC packets.
The team has overhauled MQTT integration to make topic composition more flexible and added templates to use for device and org identifiers.
The 2020.09.30.1 release is a regular blockchain maintenance update. Notably, the team has added
support for providing witness validity information to Proof of Coverage receipts, fixed blockchain
snapshot determinism and updated state channel client to make device join accepts more robust.
The 2020.09.21.1 blockchain release adds support for PoC v10.
Debug pane enhancements, ability to pause data transfer for devices and across organizations, and tools for internal monitoring by Helium dev team.
On 9/16/2020 the team activated PoC v9, which didn't go as planned, causing a complete blockchain halt for ~5 hours and subsequent slow down of blocks for ~7 hours. Longer term effects included several hotspots going out of sync for several hours after the chain resumed. Please read on for the details on what happened, what we did in the short term to fix it, and the steps we'll be taking to address these kinds of issues longer term.
Due to the recent blockchain outage with the activation of PoC v9 and extremely slow election block times, the team has agreed to revert back to PoC v8 while a more optimized mechanism to run PoC v9 is being worked upon. We will continue to post updates as we make further progress.
The 2020.09.16.0 emergency release addresses the chain halt we observed after activating PoC-v9.
The team will continue to closely monitor the situation but we're fairly confident that this update
will get the blockchain moving again.
The 2020.09.15.0 release is a major blockchain release adding support for two features: Multisig
and POC-v9.
The 2020.09.08.0 release is a minor bugfix release to fix an issue with the stabilization interval
calculations on newer Hotspots that have been loaded from a snapshot. There was an error in the how
the interval was calculated, which the PR below fixes:
On 8/12 the system changed the token allocation providing rewards for Hotspots that transferred data and required devices to pay for network traffic with Data Credits.
This release includes the ability to manually downlink from Console, UI updates, and a number of Router (backend) fixes related to packet transfer.
This 2020.09.03.0 release contains a small but significant performance improvement for state
channel causality validations for hotspots. We have also update the snapshot block to 480241.
This 2020.09.01.0 release is a minor bugfix release to fix state channel issues on the hotspots so
they can continue to transfer data. We have also added initial support for third-party gateways.
Owners of HNT can acquire Data Credits by ‘burning' their HNT into Data Credits.
In this new update, we've revamped the payments UI. Use the QR code scanner to not only populate the recipient address, but HNT amounts embedded in the QR code will populate as well!
Final Update:
The updated release had some issues with loading ledger snapshots on first-generation Hotspot
hardware, and we had to cut the 2020.08.26.0 release in order to reduce snapshot import memory
usage. Since the only new behavior was
core PR 599, we did a very short beta to
confirm the fix for the issue and released the image to GA around 12:30PM PDT.
This 2020.08.21.0 provides an implementation of
HIP-10. The
implementation is gated behind a chain var, which we will activate once everyone has upgraded and
we've finalized and announced an activation date and time.
This 2020.08.18.4 release addresses some addtional issues in the state channel client. It simply
consists of 4 commits backported from a state channel fixup branch:
Recent increases in the size of the network have caused some scaling issues. When the network was smaller, we wrote some code without concern for scaling, pulling the entirety of various data structures off of disk and into memory for the duration of various operations. The code was simpler that way, but these in-memory structures have come to cause a lot of problems as the number of gateways has gotten larger.
This 2020.08.14.1 release addresses some excessive memory use that is causing issues as the
network grows. Additionally, we have identified several improvements that should help with
performance and with the reliability of state channels.
This release fixes the websockets used on the main Accounts page (it was using the old API, now uses the new one) and updates the polling interval to 30 seconds so it can fetch recently cleared pending transactions.
It also fixes a bug where NaN is displayed for pending transactions.
This 2020.08.12.0 release addresses some of the concerns regarding the opening/closing of state
channels responsible for packet transfer. Essentially it allows the router to maintain state
channels better as we have introduced proper rollover of the channels on DC exhaustion.
This 2020.08.11.0 release fixes another out of memory condition during elections, it also fixes an
issue with transactions larger than 65kb and finally it improves restart behavior when certain
essential processes crash.
This 2020.08.10.0 release fixes an out of memory issue encountered when computing blockchain
snapshots, especially consensus members, which has been leading to the rough block times over the
weekend. Additionally a couple state channel issues have been identified and fixed as well.
This 2020.08.06.0 release fixes a few issues related to state channel v2 causality problems we
were seeing on chain. Essentially, we see hotspots file a duplicate state channel close txn despite
the fact that the router has already submitted one. This release attempts to fix that.
ICYMI next Tuesday 8/11 device owners must pay Data Credits for devices to transfer packets and Hotspots earn HNT for forwarding device data.
This release is to prep for the new Hotspots coming online, plus a few bug fixes.
This 2020.07.31.0 release fixes an issue in libp2p which prevented some hotspots from being able
to reach the router when they didn't have router in their peerbook.
This 2020.07.29.0 release fixes a bug in miner that resulted in numerous downlink LoRa packets
being dropped.
Forget Wi-Fi Network
We’ve moved the date when devices need Data Credits from 7/27 to 8/3. On 8/3 all device owners must pay Data Credits to transfer packets and Hotspots start earning HNT for data transfer from devices.
This 2020.07.24.0 release completes the "Forget Network" feature that will be added to an upcoming
version of the Helium mobile app. We also fixed a couple bugs and added debugging to prepare for
upcoming state channels V2 chain variable.
This release lands a bunch of work (4000+ lines of code) to enable state channels v2. This release lays the ground work to start charging and accounting for data credits to move data across the Helium network.
This 2020.07.16.0 release adds a chain variable for radio data gathering to aid with improvements
to proof-of-coverage. When the chain variable is activated the witness data collected will include
signal-to-noise ratio and frequency fields.
In this release, we are focused on squashing bugs and updating our community link to reflect our move to Discord!
In our last release 1.9.0, we introduced and experimental feature on a Hotspot's GPS status that
was meant to be informational.
This 2020.07.09.0 release fixes longstanding bugs in gateway-config that resulted in incorrect
WiFi network states being reported to the mobile app and via the Hotspot's LED.
For those of your running
a Miner image for Docker, there has been a
change as of miner-XXX64_2020.07.02.0_GA to make regional support easier to configure.
This release is a routine update of the onboard blessed snapshot to block height 396721. This
allows newly onboarded Helium Hotspots to sync the whole blockchain in hours not days.
The chain variable discussed here has been activated. The limit has been set at 20 for the time, being, and we'll analyze Proof of Coverage path variety and transaction validation times to see if we need to make any further adjustments.
Next week we’ll provide the ability to import devices from The Things Network Console directly from the Helium Console.
PLEASE UPDATE TO THIS RELEASE. This update includes the following new features and is not backwards
compatible. Failure to update to 1.9.0 will cause all transactions in the mobile app to fail once
we activate transaction fee chain variables.
Much of the trouble that we've been having lately with maintaining consistent block times has been a result of Hotspot clusters, some of them quite large. These clusters, often hosted in the same place, lead to an abnormally high number of witnesses and a large disk representation. Since they're colocated and witness each other, our Proof of Coverage (PoC) pathing code has some pathological behavior when they're challenged, since it must consider sometimes hundreds of Hotspots to generate the path. This can lead to single transactions that take the Hotspot hardware multiple seconds to validate, which can cause a lot of problems for protocol completion and steady block times as the validation and the Honeybadger runners compete for the disk.
Recently it's come to our attention that a growing number of Hotspots are having issues staying in sync with the chain. Hotspots use RocksDB as their storage layer for the ledger and the blockchain. RocksDB offers a feature called snapshots which purports to provide an immutable, read-only version of your store, in this case, old ledgers to make validation cheaper. Unfortunately, due to an as-yet-unidentified issue, either to do with RocksDB itself, how the RocksDB Erlang NIF library is written, or how our code is creating and using these snapshots, we're seeing some writes which have been synchronously written to disk missing from the snapshots.
While we are aware of some Hotspots falling out of sync (although they will eventually recover) we have not yet isolated the root cause. This release only makes 2 changes, it fixes a problem with the timing of LoRaWAN packets, specifically around meeting the downlink window and it blesses a new snapshot at height 379441. The new snapshot should help unstick some Hotspots that are struggling, and we will continue to work on a more permanent fix.
After monitoring the chain overnight and seeing few signs of recovery, further investigation revealed that we were still seeing out of memory errors on v1 Hotspots. A closer reading of the docs revealed that a part of the Erlang runtime reports memory usage in machine words (the width of the processor's registers), which means that the earlier measurements we based our tunings on were too low by a factor of 8. This release does nothing but retune the cache to use much less memory, which we hope will stabilize the chain, if not fully restore block times.
We've identified some more failures that have been leading to long block times and included fixes for them in this release. In addition, now we have the price oracle running, we've landed the code for real transaction fees and staking fees. These fees won't be active until we've made sure all the wallets have been updated and everyone has had appropriate warning. Finally we've bumped the blessed snapshot to 376561.
In an attempt to improve block times, we have developed a new Hotspot cache which keeps the deserialized spots in memory for longer, so we're not constantly fetching and refetching the same information from the disk. There's also a change make validation of Proof of Coverage(PoC) receipts substantially cheaper.
This minor release increases some network timeouts again to compensate for snapshot sizes. It also corrects an error in the "bad GPS fix" reporting code and makes snapshots more compatible with the upcoming changes for data credits and transaction fees. Finally there's a small fix that may help with some cases of ledger corruption. A bigger fix on this front is in the works.
See below for the updates and enhancements we’ve made to Console.
This minor release fixes a bug in the math that the price oracle uses to calculate HNT/USD.
A new app version 1.8.0 is now available for iOS and Android
This release lands support for the HNT/USD price oracle. The code will be inactive until activated via chain variable transaction, which we expect to do Friday afternoon or Monday.
As we gear up for expanding the Helium network to other regions we've been threading region awareness through our code. As of this release the Helium Network will support hotspots in the EU868 region as well as the currently supported US915 region.
Over the weekend we've identified a few minor issues with ledger snapshots, and since they're impacting block times, we're issuing a minor release to clear them up. Our old code assumed that all Hotspots would have all of the blocks, and there are a few bits of code that rely on that, one of which affects consensus nodes, which has been negatively affecting block times.
The biggest developments in the Helium blockchain ecosystem right now are around HNT and Data Credits.
Another big issue that we've been dealing with recently is sync times on new Hotspots. We've seen cases recently when new Hotspots have taken in excess of 10 days to fully sync and start issuing challenges. This release is the first to support a feature we're calling "ledger snapshots", which allow Hotspots to safely sync without downloading the entire blockchain first. A few times a day, the running consensus group will take a snapshot of its ledger, package it up with a relatively small number of blocks and then store its hash on the chain. We can then attest the validity of a particular snapshot to allow new Hotspots to download that snapshot and start using it, much as we now attest to the validity of a particular block (and all the blocks leading up to it). This should allow syncing in hours, rather than in days as is currently the case.
We activated witness refresh at block 339044 (see audit 23), which was intended to gradually clear witness tables over the course of about a week. Unfortunately we neglected to apply this refresh to the lagging ledger in the same way. Once we realized this, we issued a deactivation of the new feature at block 339291. Unfortunately it was too late and about 37 ledger entries were now divergent between the leading and lagging ledger. This led to disagreements on the block contents between the consensus members and caused a chain halt.
Today we're going to issue a transaction to change several chain variables. The high level summary:
It turns out that there is an issue with the implementation of the old witness refresh code and we're going to disable these chain variables until we can fix the issue and re-enable them.
Some common feedback we get is that Hotspots use too much bandwidth. We've taken some steps to address this in the past, and today is the latest step in scaling bandwidth back. This release applies compression to all connections between Hotspots and their other gossip peers, and makes it easier to shift to more effective compression in the future. For non-Hotspot peers, this code is backwards compatible, but you'll need to upgrade to see the benefits.
A new app version 1.7.0 is now available for iOS and Android
This release is a follow up release to 2020.05.11.0 and 2020.05.12.0 which corrects some bugs we'be observed preventing state channels from opening and fixes some scalability issues that were affecting packet routing.
This release lands a bunch of work (7000+ lines of code) to enable proper packet routing using the ledger information. This work also lays the groundwork for paying for packets with data credits. You can read more details in the earlier blog post here.
We’ve added the ability to execute functions on Console, the first can be used to decode payloads. With the Decoder Function, users can can transform and/or parse a raw payload before it is posted to an endpoint, such that, for example, individual sensor values are reported rather than the base64 string. This transformation also applies to debug output, allowing it to be more human-readable.
This release fixes some problems where nodes would re-gossip a block several times because they had not finished processing the block absorb when other copies of the block came in. This could lead to a bunch of unnecessary work as the same block would be received several times, from several peers.
This release adds support for amd64 and arm64 Docker images containing miner. These images can be fetched from Helium's image registry at quay.io. They are Alpine Linux-based so specify /bin/ash as the interactive terminal. The amd64 images will run in the cloud and the arm64 images will run on a Raspberry Pi 3 or 4 that has Ubuntu 20.04 LTS 64-bit installed.
On Tuesday, May 12th, we plan to start a network upgrade both to the blockchain and the LoRaWAN compliant infrastructure at 8 am PDT / 11 am ET. The operation is expected to be complete no later than 8:59 am PDT / 11:59 am ET. Progress will be reported live here.
We just pushed a 1.2.2 patch release of the
helium-wallet.
Lately, as our receipt count has improved, we've been seeing long block times again.
A new app version 1.6.1 is now available for iOS and Android
We identified a flaw where we did not check for 0 amount transactions in some cases. While largely harmless, it can lead to transaction spam. Additionally, after doing some load testing, we've lowered the maximum block size (measured in number of transactions) while we continue to work on improving block propogation times.
You can find the chain variable transaction details here.
We've been doing some internal work to increase transaction throughput from our API, and this has required some Hotspot side support, with respect to how the consensus group prefilters incoming transactions. The old code wasn't playing well with ordered transactions (multiple consecutive payments, for example). Expect more related fixes in the next few releases.
We just pushed a new 1.2.0 release of the
helium-wallet.
This 2020.04.13.0 release upgrades gateway-config to enable the removal of saved Wi-Fi networks
from hotspots via the mobile app. A bug fix for a recent regression that disabled GPS on the
hotspots is also included.
The new API has been deployed to api.helium.wtf (development) and api.helium.io (prod) for a couple of weeks now and as part of moving the mobile application to it we realized that two routes were not quite consistent.
This 2020.04.07.0 release adds support for multiple default LoRa routers to miner. A bug fix for
the libp2p NAT server crashing is also included. Additionally this release bumps the assumed valid
block to 282180.
Key user experience enhancements:
This 2020.04.01.0 release fixes a bug that results in hotspots getting stuck syncing. A gateway
OUI field was also added to the blockchain protocol. The new gateway OUI field is a breaking change
that requires a software upgrade.
Recently we've seen some gossip issues, where the consensus group and some small number of Hotspots manage to forge ahead on schedule, but the bulk of the Hotspots and their ancillary services fall, in lockstep, behind.
We made recent updates to Console that included backend changes and apologize if the update may have caused any disruption for existing users.
This 2020.03.26.0 release fixes a bug when validating HTLC redeems and adds a timeout to sync
processes so they do not get stuck forever. Additionally this release bumps the assumed valid block
to 263260.
A new app version 1.6.0 is now available for iOS and Android
This 2020.03.16.0 release includes a rework of the blockchain transaction manager. Additionally
this release bumps the assumed valid block to 249675.
In order to streamline device uploading to Helium Console, and, in particular, migrations from The Things Network (TTN) Console, we are releasing pre-built executables and the source code of helium-console-cli.
This 2020.03.07.0 release adds support for multi-payment transactions and includes bug fixes for
repairing the blockchain on-disk. Additionally this release bumps the assumed valid block to
234600.
In a previous release, we attempted to resolve some rare issues with consensus members forgetting they're in the consensus group. Unfortunately this traded a rare consensus member failure for a rare crash loop. This issue led to the long block over the weekend and we've been testing a fix for the last couple days.
We continue to receive excellent feedback from the community regarding some of the decisions we made with the introduction of PoC V8. This update clarifies the new chain variable transaction we issued on Thu 13 Feb 2020 09:29:39 PM UTC.
In an effort to stabilize block times the 2020.02.11.0 release adds fixes to libp2p and
blockchain-core. Additionally this release bumps the assume-valid block to 201493.
We have been steadily gathering data on the performance of Proof-of-Coverage and are ready to release the PoC version 8 update. This update brings a few key bug fixes and enhancements detailed below.
We have enabled LoRaWAN support on the Helium network. This has not been without compromises but we feel we have found a solution that should work for the vast majority of existing LoRaWAN devices.
Block time disruptions have lowered the rate of token production from the chain, and we're implementing a mechanism to shorten block times when this is the case, so that we catch up to our targeted production rates of 5 million HNT per month if we experience long block times.
Despite our prior efforts, we have still been tracking (and debugging) high memory usage in our
storage engine, RocksDB. We have identified some additional fixes and this issue has become critical
as of block 186592. This block was somewhat large and has triggered Out of Memory kills on much of
the network as it propogates out. This release contains improvements that should allow the network
to absorb past that block.
While the recent releases have done a fairly good job at cleaning up block times, the last firmware image introduced a bug that can prevent Hotspots from challenging. This release fixes that bug and should restore the normal challenge rate.
While the out of memory improvements from yesterday have helped, the erratic block times have returned. We believe this has to do with the gossip network topolology degrading over time (an OTA resets the gossip network connections as the nodes reboot). We have identified some more fixes that should help block propogation times between the consensus group which should limit the impact of bad gossip topologies on the block production rate.
After the Proof of Coverage v7 (PoC) firmware release, block times quickly began to suffer again. This time, the cause seems to be the Linux Out of Memory subsystem killing the miner process as it begins to use too much memory. We've tracked this down to RocksDB, which we use for storage for a number of subsystems, using too much memory. We applied a large number of tunings and upgraded the version of RocksDB, picking up a number of fixes and enhancements along with some memory use improvements.
A number of assert location transactions had become stalled and were replayed from the API. Unfortunately they all landed in the same few blocks, which became nearly impossible to process quickly because of some index book-keeping overhead that must be done when gateways are asserted. This release disabled some of that book-keeping, which is obsolete, and optimized other cases where the expensive work doesn't need to be done. This should allow all of the stalled Hotspots (around 5% of the fleet) to resume syncing and challenging.
Shortly, we are going to issue a chain variable which will increase the Proof of Coverage (PoC) interval from 30 blocks to 60 blocks. This will have the effect of reducing the rate of PoC requests sharply, which should decrease long blocks going forward.
We've been seeing issues of ledger corruption when nodes are syncing the chain with increased frequency and have finally identified the fix. In the way we use rocksdb we split data in the ledger and the blockchain across column families (eg token balances are distinct from hotspot information). The ledger corrruption issues were when part of a ledger update would commit but the other parts would not. The fix was to enable the atomic flush option in the rocksdb database options which ensures that all updates in a batch update are committed atomically, even across column families.
After adding an ARP mechanism to libp2p with the release of 2020.01.05.0 on January 6, 2020 we
noticed that gossip quality began to degrade again over time. To remedy this we decreased the ARP's
peer staleness time from 24 hours down to 6 minutes and began throttling ARP requests. We also
bumped the assumed valid block to 165,353 for faster resync times.
With this update we hope to alleviate some of the known concerns regarding POC path construction while also fixing some overlooked typos.
Over time we have seen our gossip quality degrade as the network has become larger. Over the
holidays, we experienced multiple blockchain stalls. Some of these stalls were the result of
out-of-memory errors. This 2020.01.05.0 release adds an ARP mechanism to libp2p to pro-actively
refresh stale peer book entries. In addition, we fixed a couple of unfortunate typos in PoC receipt
validation. We also bumped the assumed valid block to 163,853 for faster resync times.
Over time we've seen our gossip quality degrade as the network has become larger. Quite suddenly, we
crossed some threshold and gossip has become untenable without changes. libp2p groups, the
abstraction that our consensus and dkg are built upon, are taking longer and longer to converge,
leading to elections that don't complete, and occasionally to chain stalls. This emergency release
speeds up gossip dissemination by increasing the number of peers to gossip per gossip round, and
changes the seed node connection strategy to only connect to two seed nodes, rather than all of
them. At the same time, we're increasing the number of seed nodes to deal with overload.
The
PoC v5 update
exposed a few edge cases in blockchain-core related to
Uber's H3 library. The grid_distance function in H3 can crash in the
following scenarios:
With the launch and activation of PoC v5, the team quickly realized that we needed to bring back some of the PoC RSSI related probability weights back to the path construction algorithm to allow building paths with potentially known valid witnesses as next hops.
With the launch of PoC V5, we unintentionally caused an API fork, where the blockchain couldn't sync past the new election block. This happened due to the following reasons:
The launch and activation of PoC v4 introduced some interesting new paths and brought in much needed performance improvements. We have essentially quadrupled the number of challenges on the network as validation of PoC receipts is now significantly more efficient.
On December 11 around 3 PM PDT we rolled out a new beta release.
As part of our efforts to continually improve the network, we recently implemented Proof-of-Coverage (PoC) version 4. This blog attempts to detail some of the planning, decisions and analysis that went into overhauling the existing PoC mechanism.
Upon activating PoC v4 at 1:50 PM PST on December 2nd we soon realized that the calc_edge_prob had
become a performance bottleneck. In response, we changed the order of operations and switched from
using sets to lists so the calculation is less costly.
While debugging performance issues over the weekend it was realized we could significantly improve
performance of an expensive operation, calculating the ledger at a specific height, by using a
ledger snapshot we were already making. A simple cache was added to track the last 30 ledger
snapshots and if the ledger_at height existed in the cache, it will be used. This has resulted in
a significant performance improvement both to validating PoC receipts as well as block syncing.
While we still plan to activate PoCv4 tomorrow we feel that this interim performance fix, which
remains relevant for PoCv4, is worth it to get the amount of accepted PoC receipts that can be
validated and added to the chain up from the levels they have dropped to.
The prior update, shipped this morning, has had some unintented consequences. Because of a change in an init script, a latent race condition in the configuration of the WiFi interface has caused some Hotspots to be unable to connect to WiFi. Whether a Hotspot is affected depends on what firmware version it was running when it was initially configured for WiFi. Hotspots on Ethernet should not be affected.
On November 30, around 13:05 PST we forcefully selected a new consensus group since the previous consensus group got wedged because of a bad update to the hotspots on the network.
On November 30, around 10:30 PST we will be rolling out a new beta release.
On November 26 around 9:40 PM PDT we rolled out a new beta release.
On November 25 around 11:30 AM PDT we rolled out a new beta release.
We've deployed a chain var transaction to tweak some blockchain behaviour to allow for better rewards when elections run long and to try to increase the diversity of the consensus group. More details can be seen over in the chain var audit log.
On November 13 around 7am PST the blockchain halted due to an out of memory bug. All the consensus members, having agreed on the next block's contents were unable to verify them. This was because, for performance reasons, the transactions in the block were being verified in parallel. Unfortunately there was no limit on the amount of concurrency used and this led to CPU and memory exhaustion as all 65 transactions in block 110,710 were being verified simultaneously.
On November 7 around 3 PM PDT we rolled out a new beta release.
The Helium hotspot uses an ECC608 security chip which generates and stores private keys for the hotspot. The private keys generated by the ECC are not accessible outside of the secure confines of the ECC hardware.
As part of updating the blockchain-api to support newer routes and squash some critical api bugs, the team has pushed out a new release of the production blockchain-api.
We have released a firmware based on the previous beta to general availability. This firmware is focused on improving Proof of Coverage construction and validation times. The main change from the last beta has been to refactor our PoC path calculation routines to be more efficent and to improve our caching of expensive to obtain information necessary for pathing. This change can be seen at blockchain-core#289 .
On November 1 around 3 PM PDT we rolled out a new beta release to better distribute PoC requests throughout the election cycle, and including some other longer-term engineering improvements.
In preparation for release of the LongFi SDK we are testing the latest version of the LongFi protocol on a select few San Francisco hotspots that are within radio range of each other.
On October 25 around 6 PM PDT we rolled out a GA release to address blockchain syncing problems. This was a network-wide change that intentionally broke compatibility with the previous 1.0 version of the blockchain sync protocol.
As more and more hotspots ship out, we are encountering some blockchain syncing problems. A number of the shipped hotspots were assembled over a month ago but had older firmwware that did not include some important bugfixes.
The last beta deploy did not go well. Parsing of LongFi packets was broken in concentrate so the selected hotspots were not able to engage in Proof of Coverage with each other as planned.
In preparation for release of the LongFi SDK we are testing the latest version of the LongFi protocol on a select few San Francisco hotspots that are within radio range of each other.
As the number of hotspots grows with the national launch a number of issues were reported by some users and the engineering team:
We rolled out the Beta update over the weekend and it appears to have helped some, but a number of issues were discovered that caused unneeded network load, slowing down blocks and elections.
As part of improving network behavior and testing it in a real environment we use a number of opt-in beta hotspots.
The Helium blockchain team has been working for several weeks on some incremental upgrades to our Proof of Coverage (PoC) and election systems, to make them more scalable and fair. Changes here include:
