The Helium core developers have prepared a new blockchain release tagged 2021.05.16.1 which makes
a number of changes to help older Hotspots be able to sync the blockchain. This is a required
upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.
In addition, we have issued the chain variable transaction lowering the rate at which hotspots issue Proof of Coverage challenges, which should alter load network-wide.
blockchain-core/799: Change ledger startup to recover using rocks checkpoints
This PR makes some major changes to how previous versions of the ledger are stored on disk and represented in memory. A major cause of out of memory (OoM) errors in older, lower-memory hardware is the startup phase where older versions of the ledger, required for transaction validation, are interpolated from the two versions that we keep on disk. Since these interpolations live in memory, and are often quite duplicative, they consume a great deal of memory. Luckily, RocksDB, our main persistent storage engine, provides a feature called Checkpoints. While these Checkpoints were originally meant to make taking backups easier, the work here to provide lightweight copies of the ledger which remain on disk when the miner restarts, allowing us to start up with much less memory used.
Prior to this change, a subset of older, Raspberry Pi 3B+ based units with 1GB of memory were having a great deal of trouble starting up, as they would run out of memory during interpolation, and so have been stuck many days in the blockchain's past, unable to issue challenges or meaningfully participate in the network. We anticipate that this release will fix most of those units, but we remain extremely sensitive to these issues and have more performance work planned in the coming weeks.
Additionally, we've bumped the snapshot past the chain variable change mentioned above, which should help older units catch up more quickly.
Plan
We have been beta testing 2021.05.16.1 since May 16th, 2021 3:15 PM PT. Current ETA for GA is
1:30PM PT, May 17th, 2021.