The core developers have prepared an emergency release 2021.05.29.0 to fix the Mainnet chain halt
noticed around 1:00 PM PST, May 29th, 2021. This is a mandatory release for all miners on the
network. Please read further for more details.
What Happened?
The team changed the election_info API in blockchain-core as part of support for
Validators
but missed updating the relevant API calls in miner. Surprisingly xref and dialyzer, two tools
used in the Erlang ecosystem to do static analysis, didn't catch the incorrect calls either. This
regression led to a chain halt as the code change broke a critical HoneyBadgerBFT recovery code
path.
Timeline
- Sat May 29 01:04:00 PM PDT 2021: The core team got a pagerduty alert that there were no new
blocks after block
864000 - Sat May 29 01:26:00 PM PDT 2021: status updated to reflect the blockchain halt and debugging.
- Sat May 29 01:45:00 PM PDT 2021: The bug was identified as an incorrect call in miner to a critical election related function in blockchain-core, more details available in miner#797.
- Sat May 29 02:00:00 PM PDT 2021:
2021.05.29.0release candidate prepared for beta testing and imminent GA.
The issue was identified addressed quickly and the Consensus Group will be able to continue block production after this release.
Future Work
Bugs like these can slip through normal reviews and testing but improved test suites with better code coverage can help catch these sorts of regressions. Future regular maintenance releases will include these improvments.
Plan
We will begin beta testing 2021.05.29.0 as soon as the release is built and GA immediately after
checking beta status.