Skip to main content

Api Outage

· One min read

Content

With the launch of PoC V5, we unintentionally caused an API fork, where the blockchain couldn't sync past the new election block. This happened due to the following reasons:

  1. The API had been running a partially upgraded blockchain-core dependency which knew only about a subset of the new consensus rules.
  2. We issued the PoC V5 chain variables without realizing that the API would not understand how to ingest the blocks after the new rules kick in after an election happens.

To address this, we've taken the following steps:

  1. Stopped the blockchain-api writer instance.
  2. Upgraded blockchain-core dependency in blockchain-api.
  3. Issued a reset_ledger command on the blockchain-api which would bring back the correct ledger state and allow the blockchain-api to continue syncing and ingesting blocks.

Deployment Plan

The new code is running live on the blockchain-api prod instance now. We will continue to monitor its progress.