Skip to main content

Blockchain Fixes and Chain Halt Postmortem

· 2 min read

The 2020.10.27.0 blockchain release contains some critical fixes specifically meant to address the blockchain halt observed on Monday, October 26th, 2020. Please read on for a detailed breakdown.

October 26th, 2020: Chain halt postmortem

  • 11AM PST: Miner tag 2020.10.23.0 is released to GA
  • 4:30PM PST: The team notices chain stall and begins investigation
  • 5:00PM PST: Root cause identified as an unhandled EXIT error message caused while trying to create a block containing a stale poc receipt transaction. Patch deployed to the consensus group to regain block production.
  • 5:30PM PST: After having internal discussions, the team decided to roll back to previous GA tag 2020.10.12.1. It was not worth it to rush a release, especially knowing that the bug was largely harmless and the previous GA had been running solidly.
  • 6:00PM PST: All hotspots on the network downgraded back to 2020.10.12.1. By this time, network gossip caught up and block production resumed normally.

Plan

We have begun beta-testing 2020.10.27.0 around 5:15 PM PST, October 27th, 2020 and plan to GA on Thursday, October 29th, 2020 around 11AM PST.