Conduit releases Degen Chain outage report: Alerting and monitoring of Orbit chain has been improved

Conduit released a post-mortem analysis report on the previous outage of Degen Chain: On Friday, May 10 (PST time), Conduit increased the batch size of Degen and Proof of Play Apex to 10MB to reduce costs, which delayed the batch release of data from these networks to their parent chains. On Sunday, May 12, at around 1 pm, this configuration was reverted to fix the batch release. This caused a reorganization on both networks because the batch was released after the 24-hour mandatory inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay these transactions with new timestamps. After the reorganization, the node will come back with a corrupted database because its depth is not well handled by geth. This requires resynchronizing the data directory from genesis. The synchronization time for each network is more than 40 hours, and the replay rate is about 100M gas/s. Once the node is resynchronized, Conduit will try various transaction replay schemes, although not all transactions can be recovered because some transactions rely on accurate timestamps. In consultation with each rollup team, Conduit discussed and tried various strategies in parallel to bring the network online and restore the pre-reorganization state. Degen Chain went online at 7:30 pm on May 14, about 54 hours after the network was paralyzed. Proof of Play's Apex chain was restored at about the same time, but only became available to the public at 4 pm on May 15 after implementing an alternative recovery plan. In response, Conduit said it has improved alerts and monitoring of the Orbit chain to cover this situation and is committed to working with Offchain Labs to improve observability for all Orbit chain operators. The team will continue to invest and research mechanisms to better simulate mainnet conditions and transaction payloads in a test environment. Degen Chain Explorer has been displaying the latest status of Degen Chain normally.