After the Proof of Coverage v7 (PoC) firmware release, block times quickly began to suffer again. This time, the cause seems to be the Linux Out of Memory subsystem killing the miner process as it begins to use too much memory. We've tracked this down to RocksDB, which we use for storage for a number of subsystems, using too much memory. We applied a large number of tunings and upgraded the version of RocksDB, picking up a number of fixes and enhancements along with some memory use improvements.
Additionally, we've included a new garbage collector for some PoC corner cases which should help keep the ledger small and consistent from Hotspot to Hotspot.
- Tune RocksDB: Various memory-related tunings and a version upgrade. PRs: core/354, miner/297
- PoC GC: Clean up stale PoC request data in the ledger. PRs: core/307
We'll beta this release briefly starting around 11AM PST, then GA in the afternoon today, Jan 27.