In early August Nova Labs posted a change in how the denylist is operated. This change has resulted in several positive effects on the network in terms of the number of hotspots that were improved and cleared from the denylist, as well as clusters of misbehaving hotspots getting turned off, clearing innocent hotspots that were swept up in the analysis.

The ongoing challenge with the current approach is that innocent hotspots are still being swept up by the classifiers if they have especially good antenna setups since they are attractive targets for beacon replay.

Nova Labs will be deploying a refinement to the denylist that instead of denying hotspots for exhibiting improbable RF behavior denying witness events, also known as edges or connections, that are improbable. This would allow hotspots to continue to operate and earn PoC rewards for RF behavior that benefits the network overall even if they have improbable RF links.

We expect this new approach to be deployed before the end of September.

Gateway-rs, the software backing all IoT Hotspots on the Helium Network, has been upgraded to include several key performance improvements related to data delivery and Proof of Coverage response times.

On August 21, 2023, the IOT denylist will be updated to use a new weekly generation process and set of classifiers to identify hotspots that are gaming the IoT Proof of Coverage system.

The legacy Explorer (hosted at explorer-old.helium.com) and the legacy API are set for retirement on July 28. The Solana-backed Helium Explorer hosted at explorer.helium.com will continue to serve the community for coverage maps and high-level network stats.

Summary​

Previously, the Helium Foundation Console faced an exploit that was amplifying existing network usage. Following the discovery of this bug in the legacy Router/Console code backing this Network Server, several patches were deployed to prevent the exploit. While those patches were effective in fixing those specific bugs, two more bugs went overlooked and have now been patched.

Summary​

The Helium Foundation Console faced an exploit that was amplifying existing network usage and allowing bad actors to earn more than they were spending in DC. In essence, they were paying for 1 DC while the Console was burning up to 5 DC due to an accounting bug. This bug also affected all instances of Console that weren't managing their routes. This bug only affected these public instances and did not result in unpaid traffic on the Helium Network as a whole.

On 2023-04-18, a new Config Service release was deployed as part of the Oracle 1.0.0 release, which included a more rigorous request signature verification. This caused a protobuf mismatch with Packet Router and Helium Router.

Helium Wallet 1.6.0​

This latest update adds more behind the scenes support for the Solana migration, including transactions, adding, updating, and viewing Hotspots on Solana L1, and swap support.

In addition, 1.6.0 fixes a bug where users could not sign into Crowdspot and fixes the display of HNT price on iOS widgets.

The Nova developers have tagged a new 2022.12.13.0 Hotspot release for Original Helium Hotspots. This release prepares Hotspots for off-chain proof-of-coverage by updating the automatic beacon rate to every 6 hours, adds "jitter" to the beacon rate such that in the event all Hotspots come online at the same time, the PoC system is not overwhelmed with PoC.

Helium Wallet 1.5.0​

Prepare for the Helium migration to the Solana blockchain! This app update makes it easier for "blue app" users to successfully migrate to the Wallet app to manage their tokens. For those already on the Wallet app, migration will be easy. Just make sure your app is updated to the latest versions and your tokens will automatically be updated to use Solana.

What happened

On Wednesday, 2022-11-16, at approximately 22:06 UTC, Validators stopped producing blocks at block 1,619,546, and a chain halt occurred. Normal chain operations resumed 18 hours on 2022-11-17 at approximately 15:57 UTC.

The Nova developers have tagged a new 2022.11.10.0 firmware release for Original Helium Hotspots. This release updates the regional frequency plan to their latest versions as well as implements the logic required to digest band plan change.

What happened

On Wednesday, 2022-10-26, at approximately 1:57 AM UTC, Validators stopped producing blocks at block 1,587,313, and a chain halt occurred. Consequently, two days later, at 3:57 PM UTC, a second chain halt occurred at block 1,589,496. Normal chain operations resumed approximately 32 hours later at 2022-10-30 1:32 AM UTC.

New Features

Burn HNT for DC from Console​

The Wallet app now supports burn transactions from Helium Console. To burn Data Credits for packet transfer, simply tap the "Send" button and the QR code in the top left corner and scan the QR code.

The core developers have tagged a new 2022.10.28.0 miner release. This release adds exponential back-off for Hotspots when requesting ledger information from Validators. This also ensures timely back-off when block times are too high and avoids too many Hotspots attempting to connect to a Validator. The pull request specifically addresses the blockchain outages over the past week from a Hotspot's perspective.

What happened

On Wednesday 2022-10-19 at approximately 2:20 AM UTC Validators stopped producing blocks at block 1,575,935 and a chain halt occurred. Consequently, at 5:55 AM UTC a second chain halt occurred at block 1,575,955. Normal chain operations resumed approximately 5.5 hours later at 7:49 AM.

New Features

Brand New UI​

The UI has been cleaned to show a chart based on $HNT oracle price, made voting more intuitive, and improved payments on subtokens.

New Features

Import CLI Wallet​

Do you have a CLI Wallet? Do you wish you had a user interface to use instead? Now you can!

The core developers have tagged a new 2022.08.17.1 miner release. This release disables peer-to-peer activity when the Hotspot is no longer syncing the chain. This improves overall performance for the peer-to-peer network for other participants and stability of the network as a whole. Other improvements include an update to HTTP2 process that the Hotspot uses to communicate to Validators (over GRPC), and a Kerlink-specific improvement that fixes the RSSI computation if the SNR is lower than zero to reduce the number of invalid Proof-of-Coverage challenges and receipts.

What happened

On Monday 2022-08-08 at approximately 19:34 PM UTC Validators stopped producing blocks at block 1,469,951 and a chain halt occurred. Normal chain operations resumed at approximately 08:00 AM UTC just over 7 hours later.

New Features

MOBILE Support​

MOBILE support is here! Once you have MOBILE in your account, you'll be able to see its balance and send MOBILE payments to other Wallets on the network.

The team has released the latest version of Console 2.2.20 to production. This release includes a rollup of a number of updates:

The team is prioritizing its focus on issues to improve network reliability. Any new features or enhancement requests, while appreciated, will be added to our backlog and prioritized accordingly.

The core developers have tagged a new 2022.08.02.0 miner release. This release contains improvements to packet delivery such that when a Hotspot sees a packet, it sends the payload right away to the Router (and thus, recipient) immediately. Data transfer reliability is expected to improve with this release.

The team has released the latest version of Console 2.2.17 to production. This release includes a rollup of out of band releases:

The Nova developers have tagged a new 2022.07.15.0 firmware release for Original Helium Hotspots. This minor release enables mux and disables blockchain sync. Enabling mux allows the transferring of packets to use the Light Hotspot part of the codebase and is a further step forward in the Light Hotspot transition.

Sprint: 80

The team has released the latest version of Console 2.2.13 to production.

The core developers have tagged a new 2022.07.14.0 miner release. This release contains fixes to packet delivery to make it more reliable when Hotspots fall offline for a brief period of time (it will queue up to 10 device packets it saw) and fixes to let Hotspots send unknown packets if it's not synced with the blockchain.

What happened

On Monday 2022-07-11 at approximately 12:42 PM UTC the Consensus Group stopped producing blocks at block 1,435,692. Normal chain operations continued at 17:15 PM UTC just over 4 hours later.

Sprint: 79

The team has released the latest version of Console 2.2.12 to production.

New Features

Ledger Payment Support​

Users that have imported their Ledger Nano S (Android Only) or Ledger Nano X can now send payments from the app! Requires Ledger Helium App version 2.2.4. Update your Nano app using Ledger Live.

Sprint: 78

The team has released the latest version of Console 2.2.11 to production.

The core developers have tagged a new 2022.06.09.0 miner release. This release contains fixes to the packet issue identified in the last release by bumping the gateway-rs version to alpha-27, fixes a crash in ebus (more on this below), and a "flatline" fix (more on this below).

Sprint: 77

The team has released the latest version of Console 2.2.10 to production.

The core developers have tagged a new 2022.06.02.1 miner release. This is an emergency release to address issues found in 2022.06.01.0. A new dependency was introduced related to telemetry work that wasn't properly configured for the release process.

The core developers have tagged a new 2022.06.01.0 miner release. This release is mandatory for all Hotspot Makers. Contained in this release are further improvements to the flatline issue (fixes HTTP2) and enables mux. Enabling mux allows all data packet traffic to be routed through gateway-rs and away from the erlang miner. This is a key step towards removing Erlang from Hotspots altogether.

The core developers have tagged a new 2022.05.24.0 miner release. This release is mandatory for all Hotspot Makers. Contained in this release are chain variables to support Validator Performance (to be activated some time after May 25 9:00AM PT). Failure to update your fleet before the chain variables are activated will cause the Hotspot to stop functioning as Hotspots still follow the blockchain.

Sprint: 76

Despite all the excitement the team has released the latest version of Console 2.2.9 to production.

The core developers have tagged a new 2022.05.19.0 miner release. This release is highly recommended for all Hotspot Makers.

Light Hotspot Support

This update adds Light Hotspot support to the app, specifically for Transferring Hotspots and removes pre-Light Hotspot terminology from Hotspot details and Diagnostic Reports.

The core developers have tagged a new 2022.05.13.0 miner release. This release is highly recommended for all Hotspot Makers.

Bug Fixes

This update is to fix a few bugs found since the last release. The next release will address changes for Light Hotspots.

The core developers have tagged a new 2022.05.12.0 miner release. This release is highly recommended for all Hotspot Makers.

The core developers have tagged a new 2022.05.11.0 miner release. This release is highly recommended for all Hotspot Makers.

The core developers have tagged a new 2022.05.10.0 miner release. This release is highly recommended for all Hotspot Makers.

Sprint: 75

The core team has released the latest version of Console 2.2.8 to production. Changes include merging the code for XOR filter fee reduction, however it has not been activated as the vote is still ongoing. For more information check the HIP.

With HIP 55 the community approved increasing the scope of work and rewards for Validators. This document includes preliminary estimates to help provide further insight into the changes Light Hotspots have on Validator traffic.

The core developers have tagged a new 2022.04.27.0 miner release. This release is mandatory for all Hotspot Makers.

Sprint: 74

The team has released the latest version of Console 2.2.7 to production. Changes include MQTT connection overflow fixes, as well as usability and open source updates, for example payment processor Stripe is available to open source users, and new environment variables have been added to the .env template file.

The core developers have tagged a new 2022.04.19.0 miner release. This release is mandatory for all Hotspot Makers. We request that Hotspot Manufacturers upgrade their fleets by 00:00 UTC on 2022-05-03. This update contains Light Hotspot software, and when activated via chain variable on May 3 2022, will change how Hotspots receive challenges, remove block sync from Hotspots, and stabilize Hotspot performance. This will allow the approved HIP 55 chain variable to be activated and prepare for the activation of Light Hotspots.

The Nova developers have tagged a new 2022.04.14.0 firmware release for Original Helium Hotspots. This minor release increases the interval a Hotspot checks for a new miner container from 4 hours (6 times a day) to 24 hours (once per day).

Sprint: 73

The core team has released Console v2.2.6 to production. A number of updates in this sprint including increasing number of packets for multi-buy, clearer messaging for unsuccessful integrations, and a pre-built Integration with Microsoft Azure IoT Central, instructions here: https://docs.helium.com/use-the-network/console/integrations/azure/

The Nova developers have tagged a new 2022.04.05.0 firmware release for Original Helium and RAK/MNTD Hotspots.

New Features

Require Seed Words to Sign Out​

Requires users to enter their 12 word seed phrase in order to sign out to prevent accidental sign outs. Thanks to our community Mentor for writing this request!

New Features

OS Widgets (Perronef5 / Peroni_F5#3413)​

iPhone users - our community member Perroni_F5 has added an incredible feature to the Hotspot app, adding 2 widgets to your app experience:

The Nova devs have tagged a new 2022.03.31.0 firmware release for Original Helium and RAK/MNTD Hotspots.

The Helium team has tagged a new 2022.03.29.1 firmware release for Original Helium and RAK/MNTD Hotspots.

The core developers have tagged a new 2022.03.23.1 miner release. This release is mandatory for all Hotspot Makers. We request that Hotspot Manufacturers upgrade their fleets by 12:00 UTC on 2022-03-29. This will allow the approved HIP 54 chain variable to be activated and ensure Hotspots can continue creating Challenges using the new H3Dex-based PoC targeting.

The core developers are changing the PoC Interval from 475 to 950. Hotspots will PoC less often however total average rewards will remain unchanged. PoC rate has no effect on average individual Hotspot earnings, but may increase variability of rewards. This will help the chain process transactions more efficiently. Please read the details below for our reasoning.

Sprint: 72

Console v2.2.5 has been released to production. During this sprint the team also launched Helium Roaming Services. LoRaWAN network providers can learn more about how to leverage Helium coverage here.

The Helium team has tagged a new 2022.03.15.0 firmware release for Original Helium and RAK/MNTD Hotspots.

Sprint: 71

The team has released v2.2.4 release to production. A number of usability updates were completed including minimizing user input for Azure IoT Hub, making it simpler to add a label to new devices, and adding abiliity to use mobile UI to scan QR codes for devices that have implemented the LoRaWAN Device Identification QR Codes specification.

The core developers have tagged a new 2022.03.07.0 miner and blockchain-core release. This release is recommended for all Hotspot Makers.

New Features

Identify Hotspots on Denylist​

When browsing the network map on the app, Hotspots on the denylist will appear with a red badge next to their name. Denylist Hotspots are those identified to be gaming on the Helium Network.

The core developers have tagged a new 2022.02.28.0 miner and blockchain-core release. This release is recommended for all Hotspot Makers.

Sprint: 70

The latest v2.2.3 release has been releaseed to production by the team. Based on a number of conversations with users, a large number of Console API’s were added to this release and existing ones updated.

The core developers have tagged a new 2022.02.22.0 miner and blockchain-core release. This release is mandatory for all Hotspot Makers (please update your fleet by March 3, 2021 00:00 UTC) and will be mandatory in the near future for Validators, ETLs, Routers and Nodes. It contains a network improvement that is gated behind some currently inactive chain variables. Hotspots without this update will not be able to sync or mine once the chain variable is activated later.

The Helium team has tagged a new 2022.02.17.0 firmware release for Original Helium and RAK/MNTD Hotspots.

What happened

On Monday 2022-02-07 5:46 AM UTC the team announced a chain halt caused by enlarged block 121528. Normal chain operations were announced at 3:04 PM UTC, less than 10 hours later.

Sprint: 69

The team has pushed the latest v2.2.2 release to production. Highlights include updates to Receive Window Delay feature, added message queue (for Production Console only) to scale frontend database, and ux tweaks. In addition, for open source users the front end team added prebuilt Console images hosted on Quay.io so open source users do not have to build their own Console images on docker.

The Helium team has tagged a new 2022.02.10.0 firmware release for Original Helium and RAK/MNTD Hotspots. All Original Helium/RAK MNTD Hotspots have a designated hardware_type to identify its hardware version at the manufacturing stage. By targeting Hotspots by its hardware_type, this release will enable RAK/MNTD Support teams to investigate and diagnose Hotspot-related issues in a secure manner without interaction from the Helium Support team.

New Features

Transfer Hotspot v2​

A new way to transfer Hotspots is now available on the app. Now transferring Hotspots is as easy as sending a payment transaction. Hotspot sellers can initiate the transfer from Hotspot Settings, the same as before, but instead of waiting for the buyer to accept the transaction, the transfer is immediately submitted to the blockchain. Once cleared, the buyer will see the new Hotspot in their app. When the transfer is successful, both buyers and sellers will receive a notification.

The Helium team has tagged a new 2022.02.08.0 firmware release for Original Helium and RAK/MNTD Hotspots. This minor release changes the interval between automated update checks for a Hotspot's miner container. It has been updated to every 2 hours, from the current 5 minute interval. This update also changes how a Hotspot prunes old miner images, removing the possibility of re-downloading the latest image multiple times. These changes should reduce the amount of bandwidth an Original or RAK/MNTD Hotspot consumes.

Note: This release is only relevant to RAK/MNTD and Original Helium Hotspots.

Sprint: out of band

This is an out of band release that addresses the latest issue related to receive window delay, specifically this issue.

The core developers have tagged a new 2022.01.29.0 miner release. This release is recommended for all makers, but is only mandatory for Hotspots that are no longer able to load the latest snapshot. As the network gets larger and the snapshot size increases, Hotspots with less RAM will have trouble loading them. We will continue to support these Hotspots until all manufacturers have an upgrade path to Light Hotspot firmware.

The core developers have tagged a new 2022.01.27.0 miner release. This release is recommended, but not mandatory, for all Hotspot manufacturers.

Sprint: 68

This release includes mostly backend under the hood fixes. The team plans to do an out of band release next week to address the latest issue related to receive window delay, specifically this issue.

New Features

Updated Witnessed List​

Witnessed Lists in both the App and Explorer reflects the list of Hotspots that your Hotspot has seen beacon. This is an improved change from displaying a list of Hotspots that has witnessed your Hotspot as the new list directly correlates with mining rewards.

This release decouples miner releases from Hotspot firmware updates by running miner in a self-updating container.

Sprint: 67

Based on almost 5000 responses to our Console UI survey, 70% respondents reported accessing Console at least multiple times a week, and almost 50% daily.

The core developers have tagged a new 2022.01.12.1 Hotspot firmware release which includes recommended improvements to snapshot loading and reduces their memory footprint. Furthermore, this release also contains bug fixes for state channel handling specifically for the client side which should now clean the internally stored channels appropriately when they observe a state_channel_close transaction.

The core developers have tagged a new 2022.01.04.0 Hotspot firmware release which includes recommended improvements in miner specifically related to ensure that Hotspots with unasserted location are still able to sync to the head of the blockchain. Secondly this release also includes ability to use additional packet metadata provided by certain packet forwarders.

The core developers have tagged a new 2021.12.29.2 Hotspot firmware release which includes recommended improvements in blockchain-core specifically in how Snapshots are loaded and blocks are stored locally on Hotspot disks.

New Features

See how your Hotspot stacks up to the Network! We’ve added the average expected network earnings per Hotspot to see how yours perform against other Hotspots on the network. The average earnings is based on how many Hotspots are online that day, for creating challenges, beaconing, witnessing, and transferring data.

Sprint: 66.5

The latest version of Console (sprint 66.5) has been deployed to production.

Sprint: 66

The latest version of Console (sprint 66) has been deployed to production. Due to holidays we had an extended sprint 65, which shortened sprint 66. T The team also began sprint 67 which will be the last sprint of 2021 with a release planned for early January 2022.

The Helium Hotspot firmware developers have identified an issue with how the LoRaWAN packet forwarder is configured on sx1302-based Helium and RAK Hotspots after PoCv11 activation. We are issuing an emergency release to these devices to address this issue.

The core developers have tagged a new 2021.12.14.0 Hotspot firmware release which includes recommended bug fixes in erlang-libp2p and blockchain-core specifically in Snapshot loading and UPnP.

Sprint: 65

The latest version of Console (sprint 65) was released to production, including user enhancements, multi-factor authentication, and a host of backend updates.

The core developers have tagged a new 2021.12.09.0 Hotspot firmware release which includes a recommended bug fix in erlang-libp2p. This change potentially mitigates the zero-witness beacon issue that has been afflicting, mostly, relayed Challenger Hotspots failing to receive witness receipts. Special thanks to @groot#3396 and others in the #poc-discussion channel on Discord for their efforts in tracking this issue down.

Improvements

Updating Hotspot Antenna and Gain​

Several users have noted that submitting the antenna and elevation was not working as expected.

The core developers have tagged a new 2021.12.06.0 Hotspot firmware release which includes recommended fixes to Seed nodes and DNS query improvements.

Improvements

Updates Hotspot Earnings chart to show UTC days (7D, 14D, 30D).

Overview

Last week the Helium Network experienced several disruptions. Throughout the disruptions, there was no risk to HNT holders, and while some devices were unable to use the Network for data transfer, the majority of devices were unaffected.

The core developers have tagged a new 2021.11.30.1 Hotspot firmware release which includes recommended fixes to snapshot load performance, underlying rocksdb tuning and other important bug fixes.

The core developers have tagged a new 2021.11.22.0 Hotspot firmware release which includes a mandatory fix to unstick some Hotspot stuck syncing at 1,107,994 due to a nonce disagreement from loading a snapshot.

The core developers have tagged a new 2021.11.21.2 Hotspot firmware release which includes highly recommended fixes to improve network-wide block sync and snapshot ingestion on Hotspots with less memory.

The core developers have tagged a new 2021.11.19.0 Hotspot firmware release which includes highly recommended fixes to improve network-wide block sync and snapshot ingestion on Hotspots.

App Version 3.8.3

This release includes improvements to API loading, adds rewards caching, and updates how earnings are calculated.

Sprint: 64

Despite additional turbulence experienced earlier this week, the team stayed focused and released v2.1.6.

Hotspot Release​

The core developers have tagged a new 2021.11.17.2 Hotspot firmware release which includes CRITICAL fixes to ensure that Hotspots continue to follow the chain after the 24 hr+ chain halt observed on November 15th.

The core developers have tagged a new 2021.11.11.0 Hotspot firmware release which includes critical bug fixes for gossip and ARP in libp2p. We hope that this release will help Hotspots with improvements on block gossip and keeping an up to date peerbook.

The core developers have tagged a new 2021.11.10.0 Hotspot firmware release which includes critical bug fixes, supplementary code for finalizing activation of POC V11 related chain variables, packet transmission and blockchain-etl fixes.

App Version 3.8.0 (iOS) and 3.8.1 (Android)

Improvements​

• Optimizes calls to the API and cacheing
• Adds loading indicators when data calls to longer than usual
• Updates React Native - Android screenshots are no longer enabled due to a security flag that the app is required to set.
• Changes how the app retrieves chain variables from the blockchain.

The core developers have tagged a new 2021.11.04.2 Hotspot firmware release which includes a few minor usability improvements, specifically allowing Makers to access a Hotspot's region parameters via JSON-RPC and miner's CLI.

The team added another prebuilt integration with a partner akenza IoT platform, as well as updates to Configuration Profiles, and the Coverage feature.

The core developers have tagged a new 2021.11.02.0 Hotspot firmware release which includes two critical blockchain-core fixes, firstly, blockchain core can now handle v1 and v2 block gossip seamlessly; secondly, the developers noticed that the transaction_validity_version chain variable definition was set incorrectly in the header file, this has been fixed. An unfortunate effect of this is that transfer_hotspot_v2 support has been slightly delayed.

On 10/27 a fix based on a community-raised issue was deployed along with a number of other updates in the 2021.10.27.0 Hotspot firmware release.

The core developers have tagged a new 2021.10.29.0 Hotspot firmware release which includes bug fixes for dropped packets reported by the community and verified by the router team.

App Version 3.7.1

Improvements​

• updates account screen when API calls fail to avoid showing 0 balance and 0 Hotspots
• improves how the app retrieves data from the API
• fixes a bug where users couldn’t send payments or transfer Hotspot based on a security setting
• added an option to clear map cache to troubleshoot when hexagons aren’t loading or app is malfunctioning due to tileserver

The core developers have tagged a new 2021.10.27.0 Hotspot firmware release which includes new functionality for Hotspots to automatically ingest latest on-chain snapshots. Makers, please see the note below as there are some updates you'll need to incorporate to use this feature. This release also reverts the var cache which was causing unforeseen issues and upgrades blockchain-core to include many State Channel related optimizations.

App Version 3.7.0

New Features

StakeJoy as API Provider​

We've added StakeJoy API support to the Helium App. StakeJoy provides data to explorer.helium.com and now the Helium app to provide users with an improved experience. To use StakeJoy, go to Settings, tap Choose API Provider and select StakeJoy.

Included in this sprint, the team delivered Configuration Profiles provides users with a streamlined way to configure, save, and apply device settings (ADR, CF List) and a new authentication system.

The core developers have tagged a new 2021.10.18.0 Hotspot firmware release which includes some bug fixes and a new blessed snapshot at height 1059841. This release addresses a channel selection bug causing unnecessary incorrect witnesses and fixes a potential race condition when initializing var-cache. Furthermore, the updated blockchain-core has improved state channel support for router.

App Outage Post Mortem

This post-mortem describes the events leading up to the app outage on October 17, what happened, and what the team is doing moving forward.

The core developers have tagged a new 2021.10.07.0 Hotspot firmware release which includes several updates.

State channel updates to improve network performance was released to resolve a choke point caused by the exponential growth of Hotspots providing coverage. In addition, Flows has become popular to quickly send device data over The People’s Network and this release includes a number of usability updates.

App Version 3.6.0

New Features

Deployment Mode​

Ever need to hand your phone to someone for installation but don’t want them seeing your seed phrase, use the wallet payment functionality, or transfer Hotspots?

The core developers have tagged a new 2021.09.27.0 Hotspot firmware release which includes some performance fixes and a bug fix to block sync. It also includes a new core developer blessed snapshot at height 1029491. We've identified the cause of lack of agreements on on-chain snapshots and have issued a fix for it as well but this will require a validator release.

The core developers have tagged a new 2021.09.26.0 Hotspot firmware release which primarily includes networking, PoCv11 and transaction manager bug fixes. It also includes a new core developer blessed snapshot at height 1024971 while we wait for the validator pool to update to version 1.3.3.

This release includes an integration for Microsoft Azure IoT Hub. Microsoft had requested an integration with Console based on feedback from their customers. In addition, the team made usability updates based on community feedback, and enhanced device APIs.

The core developers have tagged a new 2021.09.16.1 Hotspot firmware release which primarily includes performance improvements for Hotspots. You can read about them below. This release also updates the blessed snapshot to 1012321.

The core developers have tagged a new 2021.09.14.0 Hotspot firmware release which contains several updates including support for PoCv11 (not yet activated), updates to snapshot loading, several libp2p and rewards performance changes.

A major feature the team’s worked on for the last two sprints is Coverage. Unlike Hotspot monitoring tools focused on maximizing mining rewards, Coverage is built for users of the network who want to have more visibility about which Hotspots are transferring device packets as well as displaying the status of those Hotspots all from within Console.

App Version 3.5.0

New Features

Hide Hotspot​

Have a Hotspot that has met their untimely demise and the daily reminders just reminds you of the good old days and it just hurts too much? Or maybe you want to hide it for other reasons - we don't judge. Now you can hide it from app view (but it’ll stay associated with your account). To unhide Hotspots, go to Settings to view hidden Hotspots.

The core developers have tagged a new 2021.09.03.0 Hotspot firmware release which contains a snapshot update. Please note that this tag is based off of 2021.08.31.1 and not on miner master as the team wants to continue testing PoCv11 changes that landed on master yesterday.

The core developers have tagged a new 2021.08.31.1 Hotspot firmware release which contains major blockchain and libp2p stability and performance updates. This is a mandatory release for all hotspot manufacturers to continue syncing the blockchain.

While the team focused on the Gateway Monitoring feature, this sprint they also knocked off a number of frontend and backend issues reported by the community. In addition, an out-of-band update was released last week to address a number of issues.

The core developers have tagged a new 2021.08.24.4 emergency Hotspot firmware release which contains important snapshot load fixes and a new blessed snapshot. This is a mandatory release for all hotspot manufacturers to enable the loading of new snapshots.

The core developers have tagged a new 2021.08.21.0 Hotspot firmware release which contains major blockchain-core updates specifically ensuring that snapshots are correctly getting loaded. Furthermore this release supplies a new snapshot to allow new and un-synced hotspots to continue syncing. This is mandatory release for all hotspot manufacturers.

Improvements​

• Add support for disk health in diagnostic reports
• Allow data-only Hotspots to update antenna gain and elevation

The core developers have tagged a new 2021.08.16.0 Hotspot firmware release which is a maintenance update to primarily update static seed nodes in miner. This is an optional but recommended release for Hotspot owners and manufacturers to improve network performance.

New Features​

Support for Data-Only Hotspots

• Now you can search and follow data-only Hotspots on the app, and see information specific to data-only Hotspots.

The core developers have tagged a new 2021.08.11.0 Hotspot firmware release which allows a new way to fetch seed nodes from DNS, updates the currently blessed snapshot, and contains a few other small fixes

The core developers have tagged a new 2021.08.05.0 Hotspot firmware release which updates the currently blessed snapshot and includes some fixes for state channel handling for packet routing. The release also includes support for fetching more seed nodes from DNS and some fixes for other applications using blockchain-core (router and ETL).

Improvements

• Update LongAP Descriptions
• Add Maker type to Hotspot Details*
• Add Linxdot, ClodPi, and Pisces to Hotspot Onboarding

The core developers have tagged a new 2021.08.02.0 Hotspot firmware release which updates the currently blessed snapshot and handles IPv6 addresses in bogon check. This release also removes the default Helium production router from the static configuration to help meet the demands on utilization on the network

The core developers have tagged a new 2021.07.30.0 Hotspot firmware release which updates the currently blessed snapshot, fixes a minor bug in snapshot downloads from the Amazon S3 source, and a community contribution from @PaulVMo that improves libp2p identification of private IP addresses. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

The core developers have tagged a new 2021.07.29.0 Hotspot firmware release which updates the currently blessed snapshot, adds a new seed node to the statically configured list, adjusts address hashing for Hotspot identification, and contains minor bug fixes. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

After many months of work and testing based on community feedback, the team is excited to announce the official release of Console 2.0! The team not only worked on new features and capabilities, but also focused on making sure migration to Console 2.0 would be seamless for existing users.

New Features

Fleet Mode​

• Fleet Mode is a new feature for accounts with more than 20+ Hotspots that reduces the API calls it makes and improves app performance
• Read more on Helium Docs

The core developers have tagged a new 2021.07.22.0 Hotspot firmware release which updates the currently blessed snapshot and contains networking and snapshot improvements. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

Bug Fixes​

• Improves Shortcut loading performance
• Fixes Discovery Mode modal not extending under status bar
• Fixes the “Next” button logic for Web/QR Code onboarding

The core developers have tagged a new 2021.07.19.0 Hotspot firmware release which updates the currently blessed snapshot, introduces state channel related improvements and switches all Hotspots to non-validation (follow mode). This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

After many months of work and testing based on community feedback, the team plans to release Console 2.0 on Thursday 7/29.

App Release 3.4.1

Improvements​

• Users with 15+ Hotspots in their wallet account were experiencing performance degradation. This is now fixed
• Add Web Onboarding type to support new Hotspot Makers

This release the team worked on addressing issues reported for Console 2.0 beta and also backend fixes to make joins and packet transfer more reliable.

The core developers have tagged a new 2021.07.15.0 Hotspot firmware release which introduces more snapshot related fixes. This is a mandatory release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

App Release 3.4.0

New Features​

3.4.0 Includes a small facelift to our Wallet and Hotspot tabs!

The core developers have tagged a new 2021.07.13.1 Hotspot firmware release which contains snapshot ingestion improvements and a newly blessed snapshot. This is a mandatory release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

The core developers have tagged a new 2021.07.11.0 Hotspot firmware release which contains performance improvements to snapshot ingestion. This is an optional but highly recommended release for Hotspot owners and manufacturers especially for those Hotspots which are based on older Raspberry Pi3 hardware.

The core developers have tagged a new 2021.07.09.2 Hotspot firmware release which contains a new blessed snapshot and gossip improvements. This is an optional but recommended release for Hotspot owners.

The core developers have tagged a new 2021.07.02.0 Hotspot firmware release which contains a new blessed snapshot and minor maintenance updates. This is an optional but recommended release for Hotspot owners.

This week we’re excited to release Console 2.0 beta on our staging server. Console 2.0 beta represents months of work by the team and we’re excited to share with our community and get their feedback.

The core developers have tagged a new 2021.06.30.0 Hotspot firmware release which contains updates to improve peerbook performance and a new blessed snapshot. This is an optional but recommended release for Hotspot owners.

App Release 3.3.2

New Features​

• Show Hotspot's gain and elevation

• Adds initial transaction support for staking transactions

The core developers have tagged a new 2021.06.29.0 Hotspot firmware release which primarily contains maintenance updates to potentially reduce block times and enhance block sync. This is an optional but recommended release for Hotspot owners.

The core developers prepared an emergency release 2021.06.25.3 to fix the Mainnet blockchain halt noticed around 4:00 PM PST, June 25th, 2021. This was a mandatory release for all blockchain participants on the network. Hotspot manufacturers who have not picked up this release should update their fleets. Please read further for more details.

The core developers have tagged a new 2021.06.25.0 Hotspot firmware release which contains updates to support better transaction validation for Validators and improvements to State Channel scalability. This is a required release for Hotspot owners as it introduces new chain variables which will get activated at a later date.

The core developers have tagged a new 2021.06.21.0 Hotspot firmware release which only has a new blessed snapshot update. This is an optional but recommended release for Hotspot owners; the updated blessed snapshot will help new and lagging Hotspots to sync to blockchain head faster.

App Release 3.3.1

This release fixes 2 bugs:

• Fixes a bug where RAK and Syncrob.it were not on the Hotspot onboarding list.
• Fixes a bug when the app goes in background, the privacy splash screen disappears.

The core developers have tagged a new 2021.06.18.0 Hotspot firmware release which contains network improvements related to NAT and bug fixes related to staking for Validators. This is a required release for Hotspot owners. As some of the community members have noticed that the blockchain has not generated a new snapshot since the last GA, we believe that the improvements to networking and an increment to snapshot memory generation limit will fix snapshot generation. The core developers plan to do another minor release with a snapshot update once it gets generated by the blockchain.

The team focused on delivering an pre-built integration for Microshare in addition to continuing to build out our roaming capabilities for non-Helium traffic.

App Release 3.3.0

Hexagons on the Helium Network

The core developers have tagged a new 2021.06.14.0 Hotspot firmware release which contains transaction processing improvements, preparatory code for a new chain variable, and a newly generated snapshot. This is a required release for Hotspot owners. Further announcements will be made for chain variable activation schedule but we expect about a week or so of time to allow downstream clients of the chain (etl, node, and router users) to update.

The core developers have tagged a new 2021.06.11.2 Hotspot firmware release which has minor test related fixes and a newly generated snapshot. This is an optional release for Hotspot owners but recommended to get newly onboarded and/or un-synced hotspots to sync to the head of the blockchain.

The core developers have tagged a new 2021.06.09.0 Hotspot firmware release which contains a newly generated snapshot. This would allow newly onboarded and previously un-synced hotspots to sync to the head of the blockchain much faster.

The core developers have tagged a new 2021.06.08.0 Hotspot firmware release which fixes Snapshot creation and ingestion. It also adds support for storing transaction with implicit burns. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we release to General Availability.

This sprint we included updates for downlinks including refresh and showing payloads in both base64 and decoded text.

App Release 3.2.2

Bug Fixes

• Fixes a bug where the discovery mode timer does not continue when the app is put in background
• Prevents users from submitting multiple assert location transactions before the first one clears
• Fixes onboarding errors
• Adds session length to discovery mode

As some of the community has noticed, block times and elections have been struggling the last week or so as the network continues to grow. The core developers have prepared the 2020.05.29.1 release which will adjust validation times for two known slower transactions in our blockchain. This is an optional but recommended release for miners on the network.

The core developers have prepared an emergency release 2021.05.29.0 to fix the Mainnet chain halt noticed around 1:00 PM PST, May 29th, 2021. This is a mandatory release for all miners on the network. Please read further for more details.

The core developers have tagged a new 2021.05.28.0 Hotspot firmware release which prepares support for Validators on Mainnet. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The core developers have tagged a new 2021.05.24.1 Hotspot firmware release which changes S3 snapshot download timeout to a more relaxed limit and updates the blessed snapshot to 858241. This is an optional, but recommended upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The core developers have tagged a new 2021.05.22.0 Hotspot firmware release which adds S3 download functionality for snapshots and updates the blessed snapshot to 855361. This is an optional, but recommended upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The team delivered the ability to view LoRaWAN MAC command information for downlinks. Also along with other updates backend work was done to enable Discovery Mode for Hotspots. See details about Discovery Mode here.

The core developers have tagged a new 2021.05.20.0 Hotspot firmware release which primarily fixes libp2p peerbook issues affecting a small number of Hotspots and also updates the blessed snapshot. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The team has a new release tagged 2021.05.19.0 which primarily fixes low space observed on some Hotspots with smaller SD cards and updates the blessed snapshot. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The team has a new release tagged 2021.05.18.0 which primarily fixes out-of-memory and sync issues observed on older, less capable hardware, and other bug fixes. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

Discovery Mode is Here!

Use Discovery Mode to understand your Hotspot’s coverage without waiting for Proof-of-Coverage/Beacons, or setting a location!

The Helium core developers have prepared a new blockchain release tagged 2021.05.16.1 which makes a number of changes to help older Hotspots be able to sync the blockchain. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

The team has a new release tagged 2021.05.12.0 which adds support for light gateways and miner discovery mode. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

3.1.3 Improvements and Bug Fixes

Added Relayed Hotspot pill​

Hotspots that are relayed through other Hotspots on the network may experience syncing or reduced mining rewards. We now surface that up to the user in the app.

Last night, during the rollout of the 2021.05.04.2 release including the code for HIP-29 and HIP-30, several members of the community observed a long election including multiple proposed Consensus Groups where only 15 of the 16 new members succeeded in their Distributed Key Generation (DKG). After several hours of monitoring the Consensus Group, the Decentralized Wireless Alliance (DeWi) in conjunction with several core developers ended the 450 block epoch with a Rescue Block, which later restored the network to a functioning state. The network has been operating correctly since. All funds remained safe through the entire process.

This sprint the team focused on Integration updates including building a Google Sheets prebuilt Integration, adding dynamic query parameters, and including a hold_time field to increase accuracy of timestamps.

The team has a new release tagged 2021.05.04.2 which adds support for HIP-29, HIP-30, and various bug fixes. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

3.1.1 Improvements and Bug Fixes

Updated Hotspot UI​

A modified Hotspot UI that allows for navigating to the next Hotspot in a list, without going back to the Hotspot tab.

The team has a new release tagged 2021.04.26.1 which contains some new features and bug fixes. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA. Specifically, this release includes support for Payment Memos which requires a chain variable to activate and makes this a required release. We don't intend to activate this chain variable for about a week. Please read further for more details.

This release was the first with our three-tier development environment and focused on improved downlink capabilities and provided more details for LoRaWAN MAC commands for uplink events.

3.1.0 New Features

Remote Assert​

Remote Assert is finally here! Update your Hotspot's location without being physically next to the Hotspot, and without Bluetooth pairing! This improved user experience will make it much easier for Owners and Hosts to set up Hotspots.

The team has a new release tagged 2021.04.14.0 containing an OTA Server Certificate Update.

Maker Update

Here is the latest update from our Makers manufacturing Hotspots for the Helium Network.

The team has a new release tagged 2021.04.12.0 which contains a some fixes that should allow older Hotspots to keep consuming the chain, as well as a snapshot bump.

The team has a new minor, but urgent, release tagged 2021.04.09.0 which contains a bug fix to libp2p. A latent bug was uncovered when we fixed another issue with the prior release. We are also updating the snapshot with this release.

The team has a new minor release tagged 2021.04.08.0 which contains a minor bug fix that addresses an issue that prevented snapshots from being generated by the chain. We are deploying this change today so that we can do a subsequent bump of the snapshot tomorrow. A list of other minor changes are below.

April 4-6 API Outage Retrospective​

Starting on April 4, our daily active users (DAU) on Helium Explorer doubled from 15,000 to 30,000. As a result, the API services that Explorer relies on was unable to handle the additional load. The reason for this cannot be fully attributed to DAU, but rather a symptom of the root cause. This API is used by multiple services including the Explorer, Helium App, and Wallet CLI. Many third party services also depend on this API.

The team has a new minor release tagged 2021.04.05.0 which contains a minor bug fix that caused the miner to restart every few hours, plus a snapshot bump.

The team has a new release tagged 2021.04.02.0 which contains a regular maintenance update with a support for assert_location_v2 and a snapshot bump.

The team made a number of core backend fixes related to downlinks, and initial device joins. Debug mode, and the event log also received updates.

The team has a new minor release tagged 2021.03.26.0 which contains a regular maintenance update with an improvement to the performance of distributed key generation used in consensus and a snapshot bump.

The team has a new release tagged 2021.03.24.0 which contains a significant overhaul of the Rewards transaction and maintenance improvements including a regular snapshot bump.

3.0.3 Improvements

Show which Maker paid for the Add Hotspot and Assert Location transaction

The team has a new minor release tagged 2021.03.22.0 which contains a regular maintenance update with an update to the countries supported by the miner codebase and a snapshot bump.

Maker Update​

Since the last update in February, Makers Nebra, Syncrob.it and Bobcat have concluded their Audit.

G’day mates. In addition to support for Australia 🇦🇺 (AU915), significant work to improve both the accuracy and scalability of events was completed. We also added a pre-built integration for the Tago.IO IoT platform.

The team has a new minor release tagged 2021.03.18.0 which contains a regular maintenance update with a snapshot bump and a minor update to the address hash.

RAK Hotspot Miner Firmware Issue

We have discovered an issue, this week, that affects a number of RAK Hotspot Miners shipped by Cal-Chip to customers starting January 2021 until today (the "Dolphin" batch, up to order #2999). This issue is the same as November’s discovery.

3.0.2 New Features

New Features:

1. Explore Hotspots around you if you don’t own hotspots
2. Express HNT to your local currency with CoinGecko. Tap on your HNT balance to toggle between display options. Note that data uses current $HNT.
3. Option to disable haptic feedback

The team has a new release tagged 2021.03.08.0. This release includes a fix to Hotspot firmware where onboarding keys were incorrectly generated, a snapshot bump to 750241, and several reliability fixes that were identified in our Validators Testnet.

We’re excited to announce the release of Helium Console under the Apache 2.0 license. Console joins a growing list of Helium open source projects including blockchain core, the wallet, the blockchain explorer, the Helium app and others

The team has a new release tagged 2021.03.05.0. This release collects some small but important maintenance upgrades to the blockchain but primarily is focused on adding a new functionality to the Proof of Coverage system. We also included a few code changes that will support Light Hotspots in the future and snapshot bump.

The team has a new release tagged 2021.03.02.2. This release collects several performance and maintenance upgrades to the blockchain that should improve stability and performance of members of the Consensus Group. We also included a few code changes that will support Light Hotspots in the future, a bugfix for oracle prices our testnet chain, and snapshot bump.

3.0.1 New Features

New Features:

1. See Hotspots around you

• Tap on them to view their mining details and stats

2. Show the time period when scrubbing charts on Hotspot Details

3. Make addresses compatible on the Network selectable: either copy to clipboard or open in explorer

The team has a new release tagged 2021.02.26.0. This is mostly minor from a Hotspot owner perspective but there are some major infrastructural improvements which will help us continue to maintain the code base including an upgrade to Erlang/OTP 23. We also included some region updates and a snapshot bump.

3.0.0 A Brand New App!

After months of waiting, we’re so happy to announce the new version of the Helium Hotspot app is here! Version 3.0 introduces a whole new design and introduces a new way to interact with the app. Gone are the days of scrolling for hours toiling over challenges (we have Explorer for that!).

The team has a new minor release tagged 2021.02.23.0 which contains a regular maintenance update with an update to the seed nodes and a snapshot bump.

The team has a new minor release tagged 2021.02.18.0 which contains a regular maintenance update with a snapshot bump.

This release the team added another pre-built integration with Ubidots and extended regional support for China. In addition, the team completed a number of usability enhancements, including making it easier to add Labels to devices, and identify which devices have sent data.

Maker Update​

The three newly approved Makers of the Helium network have made great progress in their audit and integration with the Hotspot app. We’d like to take this opportunity to inform the community on their progress and how long each Makers took to progress through the milestones so you, as a potential Hotspot owner, can understand when to expect your Hotspot once a Maker starts their audit process.

Contents​

The team has a new minor release tagged 2021.02.12.0 which reverts the RSSIS change from 2021.02.10.4. This change, while correct, interacts poorly with our existing PoC acceptance criteria and we're reverting it temporarily while we fix those.

Contents​

The team has a new minor release tagged 2021.02.10.4 which contains regular maintenance updates. Please read further for more details.

The team has a new minor release tagged 2021.02.09.2 which contains a reliabiliy fix for consensus group members and some CLI fixes.

The team has a new minor release tagged 2021.02.05.0 which contains an important NAT fix and a regular snapshot bump. Please read further for details.

We’re excited to launch a beta release of Flows, a new user experience designed to simplify performing Console core actions. Also, in this release notifications can be sent via webhooks, and APIs for notifications have been added.

The team has a new release tagged 2021.02.01.1 which contains minor bug fixes, routing transaction updates and a snapshot bump. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA. Please read further for more details.

The team has a new release tagged 2021.01.27.0 containing a change to the Radio Power Output, updates to RSSI offset, support for the onboarding server, bug fixes, and a snapshot bump.

2.9.0 App Update - Onboarding Support

As outlined in the blog posted January 20, 2021, the team is making a change to how Hotspots are onboarded to accommodate our new third-party Hotspots and manufacturers.

The Maker Audit

The Helium community places a lot of trust in HIP19 approved hardware vendors and grants them the ability to add full mining gateways to the Helium blockchain. These manufacturers, also known as “Makers”, are limited by chain variables and the Onboarding Server as described previously on the Helium blog. The community trusts that the Maker will produce and deliver a real physical Hotspot with approved security methods.

In this release we’ve included beta support for LoRaWAN adaptive data rate (ADR), increased the number of email notifications, and added a new API for multiple packet purchase. ADR allows devices to use an optimal data rate which reduces power consumption and airtime on the network based on RF conditions. It's recommended to only use this setting for fixed or non-mobile devices to ensure reliable connectivity.

The team has a new emergency release tagged 2021.01.17.0 which contains a bug fix to detect new RAK Hotspot Hardware.

The team has a new release tagged 2021.01.12.0 which contains minor bug fixes, test updates and a snapshot bump. Please read further for more details.

During the last sprint of 2020, the team delivered email notifications, the ability to pay for additional packets, and a number of backend updates. Configuration for email notifications and the ability to pay for additional packets has been added to Labels. This allows for flexibility over which devices trigger the notifications or receive additional packets.

The team has a new release tagged 2020.12.30.0 with improved hardware support for the RAK Hotspot Miner updates (ECC chip support), new region configs for future hotspots in China, Japan, Malaysia, Singapore and Australia, minor adjustments to challenge targeting jitter, and a snapshot bump. Please read on for more details and plan for GA.

The Helium team will be adjusting the poc_v4_target_challenge_age chain variable from 300 to 1000 in order to align more closely with the current value of poc_challenge_interval (which is currently set to 240).

The team has a new release tagged 2020.12.22.1 before the holidays with a snapshot bump and blockchain core support for showing invalid witnesses in the API. Please read on for more details and plan for GA.

The 2020.12.16.0 release is a minor blockchain maintenance release. The primary goal of this release is to bump the blessed snapshot.

The team has successfully deployed support for HIP 15 and HIP 17 and switched to a beacon-only PoC model. One of the things we decided not to change with all of the other chain variables this morning was an increase to the number of max witnesses for each PoC. Now that we are confident with the way the Consensus Group is able to handle the new PoC receipts and new rewards calculations, we feel like it's safe to increase the max number of witnesses in each PoC receipt.

The 2020.12.15.2 release is an additional bugfix release to add support for HIP15 and HIP17. Please find a detailed breakdown of the bugfixes below.

Transfer Hotspot

As approved by HIP13, the engineering team has finished the implementation of Transfer Hotspot, allowing any Hotspot owner to transfer ownership of a Hotspot to a new owner. The full help documentation can be found here: https://docs.helium.com/wallets/app-wallet/transfer-hotspot/ and CLI support here: https://developer.helium.com/blockchain/blockchain-cli#hotspots-transfer

The 2020.12.15.1 release is a bugfix release to add support for HIP17. Please find a detailed breakdown of the bugfixes below.

During this sprint we released Console backend components on our public github repo, made updates to JSON Messaging template, and made interface enhancements to help educate new users.

The 2020.12.15.2 release was a monumental release adding support for HIP17. Note that we have decided to scrap the 2020.12.09.0, 2020.12.11.0, and 2020.12.15.1 releases because of bugs we identified during ongoing testing. Details about those bug fixes can be found here and here.

With the release of HIP 16 and implementation of Random Consensus Group selection, the PoC-based Hotspot Score is no longer a metric that's used by the Helium blockchain. We intend to completely remove the score field from the API response.

The Helium blockchain team will be submitting a chain variable transaction that will increase the target interval between Proof-of-Coverage challenges (PoCs), changing the poc_challenge_interval variable from 120 to 240 blocks. This update will effectively halve the rate of PoC transactions that Hotspots generate.

This release was focused on Integration work, including adding a pre-built Adafruit IO Integration, and frontend updates to make the selected Integration more obvious.

The 2020.11.30.0 release is a minor blockchain maintenance release. The primary goal of this release is to bump the blessed snapshot.

Performance Fixes

• Migrate legacy secure store data in preparation for new app version (v3). We recommend all users update to this new app version. Failure to update to this version could mean losing access to your Wallet/Account and associated Hotspots if a user directly upgrades from 2.6.1 to 3.x.x. as a restore from the deprecated secure store will no longer be supported. This is also a good reminder to write down the 12 words used to generate the wallet's private key. You can find those 12 words by launching the Helium App > More > Advanced > Reveal 12 Words.

The team has been busy working on a new blockchain chain release tagged 2020.11.17.0. This release contains a few new features:

This release includes a new Manager user role, and other usability updates as well the Cargo tracking application was released as open source.

The team has prepared a new blockchain chain release tagged 2020.11.10.0 addressing a key issue where RAK Hotspot Miners would not properly update to new firmware OTAs.

New Features and Enhancements

Send Max HNT​

In this new update, we've made it easier for anyone to send the full amount of HNT in their wallet. You no longer have to worry about calculating the transaction fee.

The team has noticed issues where a small number of RAK Hotspot Miners are not applying firmware updates containing bug fixes.

The team has prepared a second emergency weekend release, tagged 2020.10.31.1. This release addresses a software issue that affects only a subset of the RAK Hotspot Miners, which were unable to witness challenges or build witness lists.

The team has prepared an emergency weekend release, tagged 2020.10.31.0. This release addresses a software issue that affects only the RAK Hotspot Miner where they were unable to witness challenges or build witness lists.

In this release we provide support for a second receive window, a number of UI improvements, as well as a beta feature that allows users customize JSON messages sent to Integrations.

The 2020.10.27.0 blockchain release contains some critical fixes specifically meant to address the blockchain halt observed on Monday, October 26th, 2020. Please read on for a detailed breakdown.

Update: The current release candidate primed for GA is 2020.10.23.0 which fixes minor state channel server side issues. We will keep this post updated with more details.

New Features

View Witness List​

In this new update, we've brought in a feature called "View Witnesses". This new feature can be found on your Hotspot's page. Simply toggle the View Witness to see all the Hotspots that your Hotspot has seen and interacted with!

This release includes enhancements to help manage large numbers of devices including ability to sort and customize the device table, and making group import more flexible. In addition, a number of backend fixes were made including improving join request reliability, making integrations more robust, and resolving an arbitrage issue.

The primary objective of the 2020.10.12.1 release is to address the missing data credit rewards for blockchain_txn_state_close_v1 transaction. This release also fixes inconsistent JoinAccept for packet transfers.

The primary objective of the 2020.10.08.0 release is to address the state channel overpayment exploit. The fix is guarded behind reward_version chain variable which would be incremented to 4 to activate the fix. We intend to activate this chain variable after verifying this release.

In order to comply with local North American and European regulations, we added an undocumented update to PoC to correct the spreading factor used by PoC packets.

The team has overhauled MQTT integration to make topic composition more flexible and added templates to use for device and org identifiers.

The 2020.09.30.1 release is a regular blockchain maintenance update. Notably, the team has added support for providing witness validity information to Proof of Coverage receipts, fixed blockchain snapshot determinism and updated state channel client to make device join accepts more robust.

The 2020.09.21.1 blockchain release adds support for PoC v10.

Debug pane enhancements, ability to pause data transfer for devices and across organizations, and tools for internal monitoring by Helium dev team.

On 9/16/2020 the team activated PoC v9, which didn't go as planned, causing a complete blockchain halt for ~5 hours and subsequent slow down of blocks for ~7 hours. Longer term effects included several hotspots going out of sync for several hours after the chain resumed. Please read on for the details on what happened, what we did in the short term to fix it, and the steps we'll be taking to address these kinds of issues longer term.

Due to the recent blockchain outage with the activation of PoC v9 and extremely slow election block times, the team has agreed to revert back to PoC v8 while a more optimized mechanism to run PoC v9 is being worked upon. We will continue to post updates as we make further progress.

The 2020.09.16.0 emergency release addresses the chain halt we observed after activating PoC-v9. The team will continue to closely monitor the situation but we're fairly confident that this update will get the blockchain moving again.

The 2020.09.15.0 release is a major blockchain release adding support for two features: Multisig and POC-v9.

The 2020.09.08.0 release is a minor bugfix release to fix an issue with the stabilization interval calculations on newer Hotspots that have been loaded from a snapshot. There was an error in the how the interval was calculated, which the PR below fixes:

On 8/12 the system changed the token allocation providing rewards for Hotspots that transferred data and required devices to pay for network traffic with Data Credits.

This release includes the ability to manually downlink from Console, UI updates, and a number of Router (backend) fixes related to packet transfer.

This 2020.09.03.0 release contains a small but significant performance improvement for state channel causality validations for hotspots. We have also update the snapshot block to 480241.

This 2020.09.01.0 release is a minor bugfix release to fix state channel issues on the hotspots so they can continue to transfer data. We have also added initial support for third-party gateways.

Owners of HNT can acquire Data Credits by ‘burning' their HNT into Data Credits.

New Features

Updated Payments UI​

In this new update, we've revamped the payments UI. Use the QR code scanner to not only populate the recipient address, but HNT amounts embedded in the QR code will populate as well!

Final Update:

The updated release had some issues with loading ledger snapshots on first-generation Hotspot hardware, and we had to cut the 2020.08.26.0 release in order to reduce snapshot import memory usage. Since the only new behavior was core PR 599, we did a very short beta to confirm the fix for the issue and released the image to GA around 12:30PM PDT.

This 2020.08.21.0 provides an implementation of HIP-10. The implementation is gated behind a chain var, which we will activate once everyone has upgraded and we've finalized and announced an activation date and time.

This 2020.08.18.4 release addresses some addtional issues in the state channel client. It simply consists of 4 commits backported from a state channel fixup branch:

Recent increases in the size of the network have caused some scaling issues. When the network was smaller, we wrote some code without concern for scaling, pulling the entirety of various data structures off of disk and into memory for the duration of various operations. The code was simpler that way, but these in-memory structures have come to cause a lot of problems as the number of gateways has gotten larger.

This 2020.08.14.1 release addresses some excessive memory use that is causing issues as the network grows. Additionally, we have identified several improvements that should help with performance and with the reliability of state channels.

Bug Fixes

This release fixes the websockets used on the main Accounts page (it was using the old API, now uses the new one) and updates the polling interval to 30 seconds so it can fetch recently cleared pending transactions.

It also fixes a bug where NaN is displayed for pending transactions.

This 2020.08.12.0 release addresses some of the concerns regarding the opening/closing of state channels responsible for packet transfer. Essentially it allows the router to maintain state channels better as we have introduced proper rollover of the channels on DC exhaustion.

This 2020.08.11.0 release fixes another out of memory condition during elections, it also fixes an issue with transactions larger than 65kb and finally it improves restart behavior when certain essential processes crash.

This 2020.08.10.0 release fixes an out of memory issue encountered when computing blockchain snapshots, especially consensus members, which has been leading to the rough block times over the weekend. Additionally a couple state channel issues have been identified and fixed as well.

This 2020.08.06.0 release fixes a few issues related to state channel v2 causality problems we were seeing on chain. Essentially, we see hotspots file a duplicate state channel close txn despite the fact that the router has already submitted one. This release attempts to fix that.

ICYMI next Tuesday 8/11 device owners must pay Data Credits for devices to transfer packets and Hotspots earn HNT for forwarding device data.

Bug Fixes

This release is to prep for the new Hotspots coming online, plus a few bug fixes.

• Fix crash when tapping reward transactions with 0 amount
• Add missing translation for data credit rewards
• Fix account addresses being cut off in Android transaction details
• Change hotspot activity reward transactions to just show HNT earned
• Fix payment transactions not rendering correctly in block details
• Fix assert location for accounts with 0 balance

Summary​

• We're moving the Phase 3 Data Credits go live date to 8/11 for further testing and fixing issues.
• A fix has been pushed to address issue with Hotspots not transferring packets.
• We're going to improve communication and status of issues with this tool, subscribe here.

This 2020.07.31.0 release fixes an issue in libp2p which prevented some hotspots from being able to reach the router when they didn't have router in their peerbook.

This 2020.07.29.0 release fixes a bug in miner that resulted in numerous downlink LoRa packets being dropped.

New Features

Forget Wi-Fi Network

• This feature allows users to forget a wi-fi network that the Hotspot has remembered, so that you can either update the password, or connect to a new network
• To forget a network, tap the Hotspot settings, pair with the Hotspot, and tap the network name. The app will cycle through all the networks profiles saved on the Hotspot until they're all forgotten. Then you can connect to a new network.

We’ve moved the date when devices need Data Credits from 7/27 to 8/3. On 8/3 all device owners must pay Data Credits to transfer packets and Hotspots start earning HNT for data transfer from devices.

This 2020.07.24.0 release completes the "Forget Network" feature that will be added to an upcoming version of the Helium mobile app. We also fixed a couple bugs and added debugging to prepare for upcoming state channels V2 chain variable.

This release lands a bunch of work (4000+ lines of code) to enable state channels v2. This release lays the ground work to start charging and accounting for data credits to move data across the Helium network.

This 2020.07.16.0 release adds a chain variable for radio data gathering to aid with improvements to proof-of-coverage. When the chain variable is activated the witness data collected will include signal-to-noise ratio and frequency fields.

In this release, we are focused on squashing bugs and updating our community link to reflect our move to Discord!

Key feature updates​

• Acquire Data Credits directly from Console.
• Import multiple devices from The Things Network Console directly from the Helium Console.

Removal of GPS Status​

In our last release 1.9.0, we introduced and experimental feature on a Hotspot's GPS status that was meant to be informational.

This 2020.07.09.0 release fixes longstanding bugs in gateway-config that resulted in incorrect WiFi network states being reported to the mobile app and via the Hotspot's LED.

For those of your running a Miner image for Docker, there has been a change as of miner-XXX64_2020.07.02.0_GA to make regional support easier to configure.

This release is a routine update of the onboard blessed snapshot to block height 396721. This allows newly onboarded Helium Hotspots to sync the whole blockchain in hours not days.

The chain variable discussed here has been activated. The limit has been set at 20 for the time, being, and we'll analyze Proof of Coverage path variety and transaction validation times to see if we need to make any further adjustments.

Key feature updates​

Next week we’ll provide the ability to import devices from The Things Network Console directly from the Helium Console.

Summary

PLEASE UPDATE TO THIS RELEASE. This update includes the following new features and is not backwards compatible. Failure to update to 1.9.0 will cause all transactions in the mobile app to fail once we activate transaction fee chain variables.

Much of the trouble that we've been having lately with maintaining consistent block times has been a result of Hotspot clusters, some of them quite large. These clusters, often hosted in the same place, lead to an abnormally high number of witnesses and a large disk representation. Since they're colocated and witness each other, our Proof of Coverage (PoC) pathing code has some pathological behavior when they're challenged, since it must consider sometimes hundreds of Hotspots to generate the path. This can lead to single transactions that take the Hotspot hardware multiple seconds to validate, which can cause a lot of problems for protocol completion and steady block times as the validation and the Honeybadger runners compete for the disk.

Recently it's come to our attention that a growing number of Hotspots are having issues staying in sync with the chain. Hotspots use RocksDB as their storage layer for the ledger and the blockchain. RocksDB offers a feature called snapshots which purports to provide an immutable, read-only version of your store, in this case, old ledgers to make validation cheaper. Unfortunately, due to an as-yet-unidentified issue, either to do with RocksDB itself, how the RocksDB Erlang NIF library is written, or how our code is creating and using these snapshots, we're seeing some writes which have been synchronously written to disk missing from the snapshots.

While we are aware of some Hotspots falling out of sync (although they will eventually recover) we have not yet isolated the root cause. This release only makes 2 changes, it fixes a problem with the timing of LoRaWAN packets, specifically around meeting the downlink window and it blesses a new snapshot at height 379441. The new snapshot should help unstick some Hotspots that are struggling, and we will continue to work on a more permanent fix.

After monitoring the chain overnight and seeing few signs of recovery, further investigation revealed that we were still seeing out of memory errors on v1 Hotspots. A closer reading of the docs revealed that a part of the Erlang runtime reports memory usage in machine words (the width of the processor's registers), which means that the earlier measurements we based our tunings on were too low by a factor of 8. This release does nothing but retune the cache to use much less memory, which we hope will stabilize the chain, if not fully restore block times.

We've identified some more failures that have been leading to long block times and included fixes for them in this release. In addition, now we have the price oracle running, we've landed the code for real transaction fees and staking fees. These fees won't be active until we've made sure all the wallets have been updated and everyone has had appropriate warning. Finally we've bumped the blessed snapshot to 376561.

In an attempt to improve block times, we have developed a new Hotspot cache which keeps the deserialized spots in memory for longer, so we're not constantly fetching and refetching the same information from the disk. There's also a change make validation of Proof of Coverage(PoC) receipts substantially cheaper.

This minor release increases some network timeouts again to compensate for snapshot sizes. It also corrects an error in the "bad GPS fix" reporting code and makes snapshots more compatible with the upcoming changes for data credits and transaction fees. Finally there's a small fix that may help with some cases of ledger corruption. A bigger fix on this front is in the works.

Key feature updates​

See below for the updates and enhancements we’ve made to Console.

This minor release fixes a bug in the math that the price oracle uses to calculate HNT/USD.

Summary​

A new app version 1.8.0 is now available for iOS and Android

This release lands support for the HNT/USD price oracle. The code will be inactive until activated via chain variable transaction, which we expect to do Friday afternoon or Monday.

As we gear up for expanding the Helium network to other regions we've been threading region awareness through our code. As of this release the Helium Network will support hotspots in the EU868 region as well as the currently supported US915 region.

Over the weekend we've identified a few minor issues with ledger snapshots, and since they're impacting block times, we're issuing a minor release to clear them up. Our old code assumed that all Hotspots would have all of the blocks, and there are a few bits of code that rely on that, one of which affects consensus nodes, which has been negatively affecting block times.

The biggest developments in the Helium blockchain ecosystem right now are around HNT and Data Credits.

Another big issue that we've been dealing with recently is sync times on new Hotspots. We've seen cases recently when new Hotspots have taken in excess of 10 days to fully sync and start issuing challenges. This release is the first to support a feature we're calling "ledger snapshots", which allow Hotspots to safely sync without downloading the entire blockchain first. A few times a day, the running consensus group will take a snapshot of its ledger, package it up with a relatively small number of blocks and then store its hash on the chain. We can then attest the validity of a particular snapshot to allow new Hotspots to download that snapshot and start using it, much as we now attest to the validity of a particular block (and all the blocks leading up to it). This should allow syncing in hours, rather than in days as is currently the case.

What Happened?​

We activated witness refresh at block 339044 (see audit 23), which was intended to gradually clear witness tables over the course of about a week. Unfortunately we neglected to apply this refresh to the lagging ledger in the same way. Once we realized this, we issued a deactivation of the new feature at block 339291. Unfortunately it was too late and about 37 ledger entries were now divergent between the leading and lagging ledger. This led to disagreements on the block contents between the consensus members and caused a chain halt.

Today we're going to issue a transaction to change several chain variables. The high level summary:

It turns out that there is an issue with the implementation of the old witness refresh code and we're going to disable these chain variables until we can fix the issue and re-enable them.

Some common feedback we get is that Hotspots use too much bandwidth. We've taken some steps to address this in the past, and today is the latest step in scaling bandwidth back. This release applies compression to all connections between Hotspots and their other gossip peers, and makes it easier to shift to more effective compression in the future. For non-Hotspot peers, this code is backwards compatible, but you'll need to upgrade to see the benefits.

Summary​

A new app version 1.7.0 is now available for iOS and Android

This release is a follow up release to 2020.05.11.0 and 2020.05.12.0 which corrects some bugs we'be observed preventing state channels from opening and fixes some scalability issues that were affecting packet routing.

Key feature updates​

This release lands a bunch of work (7000+ lines of code) to enable proper packet routing using the ledger information. This work also lays the groundwork for paying for packets with data credits. You can read more details in the earlier blog post here.

Key feature updates​

Functions​

We’ve added the ability to execute functions on Console, the first can be used to decode payloads. With the Decoder Function, users can can transform and/or parse a raw payload before it is posted to an endpoint, such that, for example, individual sensor values are reported rather than the base64 string. This transformation also applies to debug output, allowing it to be more human-readable.

This release fixes some problems where nodes would re-gossip a block several times because they had not finished processing the block absorb when other copies of the block came in. This could lead to a bunch of unnecessary work as the same block would be received several times, from several peers.

This release adds support for amd64 and arm64 Docker images containing miner. These images can be fetched from Helium's image registry at quay.io. They are Alpine Linux-based so specify /bin/ash as the interactive terminal. The amd64 images will run in the cloud and the arm64 images will run on a Raspberry Pi 3 or 4 that has Ubuntu 20.04 LTS 64-bit installed.

On Tuesday, May 12th, we plan to start a network upgrade both to the blockchain and the LoRaWAN compliant infrastructure at 8 am PDT / 11 am ET. The operation is expected to be complete no later than 8:59 am PDT / 11:59 am ET. Progress will be reported live here.

We just pushed a 1.2.2 patch release of the helium-wallet.

Block Weather​

Recently, we mentioned that we had some pending speedups to our HBBFT implementation. This image releases them to the cluster.

Block Weather​

While yesterdays fix for block times (see here) went well, a seemingly unrelated change has caused quite a bit of trouble today.

Block Weather​

Lately, as our receipt count has improved, we've been seeing long block times again.

Summary​

A new app version 1.6.1 is now available for iOS and Android

Summary​

We identified a flaw where we did not check for 0 amount transactions in some cases. While largely harmless, it can lead to transaction spam. Additionally, after doing some load testing, we've lowered the maximum block size (measured in number of transactions) while we continue to work on improving block propogation times.

Audit​

You can find the chain variable transaction details here.

Key feature updates​

• We’re releasing Debug mode Monday April 20 as we described last week.
• Users can access Debug mode on a specific device or label page. This allows users to view data from a single or multiple devices.
• Debug mode will display 10 packets at a time.
• Each packet is made up of two parts: the information sent from Console to an integration, and information sent to Console from the backend independent of the integration.

We've been doing some internal work to increase transaction throughput from our API, and this has required some Hotspot side support, with respect to how the consensus group prefilters incoming transactions. The old code wasn't playing well with ordered transactions (multiple consecutive payments, for example). Expect more related fixes in the next few releases.

We just pushed a new 1.2.0 release of the helium-wallet.

Summary​

This 2020.04.13.0 release upgrades gateway-config to enable the removal of saved Wi-Fi networks from hotspots via the mobile app. A bug fix for a recent regression that disabled GPS on the hotspots is also included.

Key feature updates​

• AWS IoT downlink completed, more details about documentation here.
• This week pushed out backend updates: see previous blog here.
• Dedudped data sent from Helium Hotspots.

The new API has been deployed to api.helium.wtf (development) and api.helium.io (prod) for a couple of weeks now and as part of moving the mobile application to it we realized that two routes were not quite consistent.

Summary​

This 2020.04.07.0 release adds support for multiple default LoRa routers to miner. A bug fix for the libp2p NAT server crashing is also included. Additionally this release bumps the assumed valid block to 282180.

Summary​

Key user experience enhancements:

• Labels mapped to integrations are easily identifiable by Integration icon.
• Device details: includes packet transferred numbers.
• Device index page includes: Labels, Integration, Frames up, Frames down, Date added, Last connected.

Summary​

This 2020.04.01.0 release fixes a bug that results in hotspots getting stuck syncing. A gateway OUI field was also added to the blockchain protocol. The new gateway OUI field is a breaking change that requires a software upgrade.

Block Weather​

Recently we've seen some gossip issues, where the consensus group and some small number of Hotspots manage to forge ahead on schedule, but the bulk of the Hotspots and their ancillary services fall, in lockstep, behind.

Summary​

We made recent updates to Console that included backend changes and apologize if the update may have caused any disruption for existing users.

Summary​

This 2020.03.26.0 release fixes a bug when validating HTLC redeems and adds a timeout to sync processes so they do not get stuck forever. Additionally this release bumps the assumed valid block to 263260.

Block Weather​

For that past month or so, our various fixes and optimizations have done a fairly good job of preventing long blocks and keeping token emissions on schedule. For more details see here and here.

Summary​

A new app version 1.6.0 is now available for iOS and Android

Summary​

This 2020.03.16.0 release includes a rework of the blockchain transaction manager. Additionally this release bumps the assumed valid block to 249675.

Summary​

In order to streamline device uploading to Helium Console, and, in particular, migrations from The Things Network (TTN) Console, we are releasing pre-built executables and the source code of helium-console-cli.

Summary​

This 2020.03.07.0 release adds support for multi-payment transactions and includes bug fixes for repairing the blockchain on-disk. Additionally this release bumps the assumed valid block to 234600.

Summary​

In a previous release, we attempted to resolve some rare issues with consensus members forgetting they're in the consensus group. Unfortunately this traded a rare consensus member failure for a rare crash loop. This issue led to the long block over the weekend and we've been testing a fix for the last couple days.

Summary​

We continue to receive excellent feedback from the community regarding some of the decisions we made with the introduction of PoC V8. This update clarifies the new chain variable transaction we issued on Thu 13 Feb 2020 09:29:39 PM UTC.

Block Weather​

In an effort to stabilize block times the 2020.02.11.0 release adds fixes to libp2p and blockchain-core. Additionally this release bumps the assume-valid block to 201493.

Summary​

We have been steadily gathering data on the performance of Proof-of-Coverage and are ready to release the PoC version 8 update. This update brings a few key bug fixes and enhancements detailed below.

Executive summary​

We have enabled LoRaWAN support on the Helium network. This has not been without compromises but we feel we have found a solution that should work for the vast majority of existing LoRaWAN devices.

Executive summary​

Block time disruptions have lowered the rate of token production from the chain, and we're implementing a mechanism to shorten block times when this is the case, so that we catch up to our targeted production rates of 5 million HNT per month if we experience long block times.

Block Weather​

Despite our prior efforts, we have still been tracking (and debugging) high memory usage in our storage engine, RocksDB. We have identified some additional fixes and this issue has become critical as of block 186592. This block was somewhat large and has triggered Out of Memory kills on much of the network as it propogates out. This release contains improvements that should allow the network to absorb past that block.

Block Weather​

While the recent releases have done a fairly good job at cleaning up block times, the last firmware image introduced a bug that can prevent Hotspots from challenging. This release fixes that bug and should restore the normal challenge rate.

Block Weather​

While the out of memory improvements from yesterday have helped, the erratic block times have returned. We believe this has to do with the gossip network topolology degrading over time (an OTA resets the gossip network connections as the nodes reboot). We have identified some more fixes that should help block propogation times between the consensus group which should limit the impact of bad gossip topologies on the block production rate.

Block Weather​

After the Proof of Coverage v7 (PoC) firmware release, block times quickly began to suffer again. This time, the cause seems to be the Linux Out of Memory subsystem killing the miner process as it begins to use too much memory. We've tracked this down to RocksDB, which we use for storage for a number of subsystems, using too much memory. We applied a large number of tunings and upgraded the version of RocksDB, picking up a number of fixes and enhancements along with some memory use improvements.

Block Weather​

As you may have noticed, block times recently have regularly not been meeting their typical sixty second deadline with any regularity. You can read more about how we deal with slow block times and long elections here.

A number of assert location transactions had become stalled and were replayed from the API. Unfortunately they all landed in the same few blocks, which became nearly impossible to process quickly because of some index book-keeping overhead that must be done when gateways are asserted. This release disabled some of that book-keeping, which is obsolete, and optimized other cases where the expensive work doesn't need to be done. This should allow all of the stalled Hotspots (around 5% of the fleet) to resume syncing and challenging.

Plan​

Shortly, we are going to issue a chain variable which will increase the Proof of Coverage (PoC) interval from 30 blocks to 60 blocks. This will have the effect of reducing the rate of PoC requests sharply, which should decrease long blocks going forward.

We've been seeing issues of ledger corruption when nodes are syncing the chain with increased frequency and have finally identified the fix. In the way we use rocksdb we split data in the ledger and the blockchain across column families (eg token balances are distinct from hotspot information). The ledger corrruption issues were when part of a ledger update would commit but the other parts would not. The fix was to enable the atomic flush option in the rocksdb database options which ensures that all updates in a batch update are committed atomically, even across column families.

After adding an ARP mechanism to libp2p with the release of 2020.01.05.0 on January 6, 2020 we noticed that gossip quality began to degrade again over time. To remedy this we decreased the ARP's peer staleness time from 24 hours down to 6 minutes and began throttling ARP requests. We also bumped the assumed valid block to 165,353 for faster resync times.

With this update we hope to alleviate some of the known concerns regarding POC path construction while also fixing some overlooked typos.

Over time we have seen our gossip quality degrade as the network has become larger. Over the holidays, we experienced multiple blockchain stalls. Some of these stalls were the result of out-of-memory errors. This 2020.01.05.0 release adds an ARP mechanism to libp2p to pro-actively refresh stale peer book entries. In addition, we fixed a couple of unfortunate typos in PoC receipt validation. We also bumped the assumed valid block to 163,853 for faster resync times.

Over time we've seen our gossip quality degrade as the network has become larger. Quite suddenly, we crossed some threshold and gossip has become untenable without changes. libp2p groups, the abstraction that our consensus and dkg are built upon, are taking longer and longer to converge, leading to elections that don't complete, and occasionally to chain stalls. This emergency release speeds up gossip dissemination by increasing the number of peers to gossip per gossip round, and changes the seed node connection strategy to only connect to two seed nodes, rather than all of them. At the same time, we're increasing the number of seed nodes to deal with overload.

The PoC v5 update exposed a few edge cases in blockchain-core related to Uber's H3 library. The grid_distance function in H3 can crash in the following scenarios: