The 2020.09.16.0
emergency release addresses the chain halt we observed after activating PoC-v9.
The team will continue to closely monitor the situation but we're fairly confident that this update
will get the blockchain moving again.
Contents
With every chain variable update there's a little uncertainty that older code paths may get broken. The team tries their best to test any such changes thoroughly but sometimes bugs slip through the cracks.
What exactly happened?
-
We believe that the root cause of the chain halt was the fact that we were re-validating
poc_receipts_txn
by using a lagging ledger while trying to absorb the transaction. We have addressed that by getting the required channel information for witnesses by walking the chain backwards instead of relying on a potentially old ledger. -
Furthermore,
rewards_txn
were being extremely expensive due to unnecessary logging, thereby causing IO issues. We have reduced the log levels frominfo
todebug
to address that.
Related PRs
-
- Fix poc eunits and reduce log level
-
- Dont call check_is_valid_poc to get channels*
Plan
Since this is an emergency release, we have already began betaing it on some hotspots and will be deploying it throughout the network around 5:30PM PST, Wednesday 16th Sept.