According to Blockworks, a post-mortem report on the recent Solana network outage revealed that a previously known bug was responsible for the issue. The network was down for five hours on Tuesday, with validators, a crucial infrastructure layer of Solana, restarting the network before 10 am ET. The post-mortem, authored by Anza, determined that an infinite loop caused by a bug led to validators stalling on a specific block. This resulted in validators endlessly repeating an action, preventing the network from processing more transactions.
Solana's Just-in-Time (JIT) compilation cache encountered a bug, causing an infinite loop of recompilations for some older programs, monopolizing network resources and halting operations. Consequently, the compiler, designed to improve performance by compiling code in real time, became a bottleneck. The Solana team deployed a fix in a new release to address the bug and prevent such loops. The issue was consistent with a bug identified during the investigation of a recent Devnet outage, and a patch was set to be deployed imminently.
The problem was first reported to the Solana security team in April 2022. The report explained that the fix eliminates the ability to create the preconditions required to trigger the bug, offering a simpler resolution. A more comprehensive fix will be included with further improvements to LoadedPrograms and allowed to stabilize with the regular release cycle.
This outage marks the sixth major outage for Solana since its inception. The last outage occurred in February of last year. The price of SOL, Solana's native asset, briefly dipped on the news of the outage but quickly recovered. Market data suggests that market responses to Solana network outages have become more muted over time.
