Blockchain Fixes and Chain Halt Postmortem

The 2020.10.27.0 blockchain release contains some critical fixes specifically meant to address the blockchain halt observed on Monday, October 26th, 2020. Please read on for a detailed breakdown.

October 26th, 2020: Chain halt postmortem

• 11AM PST: Miner tag 2020.10.23.0 is released to GA
• 4:30PM PST: The team notices chain stall and begins investigation
• 5:00PM PST: Root cause identified as an unhandled EXIT error message caused while trying to create a block containing a stale poc receipt transaction. Patch deployed to the consensus group to regain block production.
• 5:30PM PST: After having internal discussions, the team decided to roll back to previous GA tag 2020.10.12.1. It was not worth it to rush a release, especially knowing that the bug was largely harmless and the previous GA had been running solidly.
• 6:00PM PST: All hotspots on the network downgraded back to 2020.10.12.1. By this time, network gossip caught up and block production resumed normally.

Highlights and Related PRs

• Better transaction error handling and testing. blockchain-core#665
• Enhance client side state channel validation. blockchain-core#667
• Fix state channel tests. blockchain-core#668
• Add EU868 region for Ukraine. miner#550
• Fix argument order for LoRaWAN rx2 window. miner#551
• Update blockchain-core and snapshot block to 560881. miner#552

Plan

We have begun beta-testing 2020.10.27.0 around 5:15 PM PST, October 27th, 2020 and plan to GA on Thursday, October 29th, 2020 around 11AM PST.