Solana Foundation Still Doesn’t Know What Caused Weekend Network Outage

Two days after a major outage, the Solana Foundation says it’s still investigating why the Solana network went down for almost 20 hours following a network upgrade.

“The cause of this is still unknown and under active investigation,” the Foundation wrote in a blog post published on Sunday night. A Solana Foundation spokesperson told Decrypt in an email on Monday that there are still no updates to the information that’s already been shared in its blog.

That’s a big blow for a network that’s become a major player in the industry in just two years.

Solana is the ninth-largest blockchain by the total value of assets on the network, $551 million at the time of writing, according to CoinMarketCap. It launched in 2020 as a faster, cheaper alternative to Ethereum. Since then, it’s become home to the second-largest NFT market in the industry, having seen $2.6 million in sales in the past day alone, according to CryptoSlam.

It’s also home to a small, yet growing DeFi ecosystem—the kinds of tools that enable non-custodial trading, lending, and borrowing, all done on-chain and without third-party intermediaries. The DeFi community on Solana recently took a big hit when crypto exchange FTX filed for bankruptcy and the team behind Serum, a decentralized exchange, had to scramble to shutdown the project founded by Sam Bankman-Fried. As of this writing, DeFi protocols on Solana account for $108 million in total value locked, according to DeFi Llama.

During the outage, which started just before 6 a.m. UTC on Saturday, the network was unable to process user transactions. This means all on-chain activity, including NFT and DeFi trading, came to a screeching halt. After engineers recommended restarting the network, validators had to downgrade to an earlier version of software used to run nodes. They restarted the network by 2 a.m. UTC on Sunday.

On Saturday, the price of Solana’s token, SOL, took a hit on the news that the network was experiencing an outage. It started the day trading at $23.03, but fell 6% to $21.71 before the outage ended on Sunday morning, according to CoinGecko. By Monday afternoon, SOL had almost made up the outage-induced losses and was trading at $22.46.

But backlash on Twitter was fiery and persistent through the weekend.

“DeFi doesn’t work on a chain that goes down, no matter how low the fees are,” wrote a user who goes by 0xShitTrader on Twitter and says they’re an office manager for Ellipsis Labs.

The company has been building Phoenix, a decentralized limit order book, on Solana and has previously said it was attracted to building on the network because of its high throughput and low transaction fees.

The outage also brought out some heavy criticism from Paul Brody, who heads up blockchain initiatives at global accounting firm EY.

“Solana is like a lifestyle-friendly blockchain because you can have nights and weekends off,” Brody wrote on Twitter Saturday. “When can we all just admit that Solana is a joke. We’re so far beyond farce here. How can you [build] mission critical infrastructure on this nonsense,” he continued in another tweet.

Solana co-founder Anatoly Yakovenko said on an episode of Decrypt’s gm podcast last year that outages are “the biggest challenge for us, and the number one priority.” At the time, he said he saw a new validator client, Firedancer, as a “long-term fix.”

The client has a built-in fail-safe that switches validators to “vote-only” mode in the event of an outage. That means the network prioritizes voting transactions, which are needed to restart the network, over regular user transactions.

Saturday’s Solana outage is the latest in a string of issues the Solana network has faced. From September 30 until October 1, the network saw degraded performance that turned into a 7-hour outage.

The Saturday outage came on the heels of unrelated news about Solana Spaces, two “IRL” stores in New York and Miami, that opened in July and are now shutting down.

Vibhu Norby, CEO and founder of Solana Spaces, announced the news last week on Twitter, saying that the initiative would “pivot our Solana onboarding efforts into digital products like DRiP, our free NFT product with more than 100k sign-ups.”

Of course, free NFT drops only work as an onboarding strategy if the chain stays online.