The 2020.09.16.0
emergency release addresses the chain halt we observed after activating PoC-v9.
The team will continue to closely monitor the situation but we're fairly confident that this update
will get the blockchain moving again.
Contents
With every chain variable update there's a little uncertainty that older code paths may get broken. The team tries their best to test any such changes thoroughly but sometimes bugs slip through the cracks.
What exactly happened?
We believe that the root cause of the chain halt was the fact that we were re-validating
poc_receipts_txn
by using a lagging ledger while trying to absorb the transaction. We have addressed that by getting the required channel information for witnesses by walking the chain backwards instead of relying on a potentially old ledger.Furthermore,
rewards_txn
were being extremely expensive due to unnecessary logging, thereby causing IO issues. We have reduced the log levels frominfo
todebug
to address that.
Related PRs
- Fix poc eunits and reduce log level
- Dont call check_is_valid_poc to get channels*
Plan
Since this is an emergency release, we have already began betaing it on some hotspots and will be deploying it throughout the network around 5:30PM PST, Wednesday 16th Sept.