Ethereum developers will be holding their breath for the 12 minutes it takes to work out if Ethereum’s Pectra hard fork has finalized properly after it’s deployed tomorrow.

During every Ethereum hard fork, there is roughly a 12-minute period — the time it takes for two epochs — where over a hundred tireless Ethereum developers vigilantly monitor the state of the network for signs of problems, wondering, Did we do enough?

Every time Ethereum goes through a major hard fork — which is like replacing components of an airplane in mid-flight — there’s a niggling chance that something could go horribly wrong. 

But for a decade now, devs have made sure that has never happened — despite some notable problems following the Shanghai fork in 2023 and the Altair fork in 2021.

“We’ve had 10 years without a liveness failure. This is one of the most important parts of the protocol,” Mallesh Pai, a senior director of research at Ethereum software and infrastructure firm Consensys, tells Magazine. 

“It’s gone without missing a beat, and through some incredible changes, like the Merge, which was a gigantic feat of engineering to literally change the underlying consensus and still not skip a beat.”

“You build that trust over 10 years. If you mess it up once, you’re back to zero like everybody else.”

Ethereum’s upcoming major hard fork — the Pectra upgrade set for May 7 at 10:05 am UTC — has had an unusually troubled path to mainnet. For the largest hard fork in the blockchain’s history, it’s taken a year and a half to get to this point and failed to finalize properly on the first two testnets. And for the first time in Ethereum’s history, it required a third testnet to be set up in direct response to the issues. 

So, what exactly happens in those 12 minutes? 

One epoch is roughly 6.4 minutes, the period of time it takes for a random committee of Ethereum validators to propose and vote on the validity of new blocks. It’s the time taken for up to 32 blocks to be added to the chain. The first two to three epochs are the danger zone where any issues from a new upgrade are likely to rear their ugly heads. 

“The minute they go up to the fork […] they’re not stressed,” explains Nixo, a member of the Ethereum Foundation’s protocol support team. “But it’s like, the one, two, 12 minutes after the fork where they’re watching for finalization, making sure none of the clients are forking off onto other chains or going off onto other forks.”

“So, I would say it’s, like, the 15 minutes after the fork happens that there’s sort of like [gasp] holding your breath.”

During this period, developers are anxiously checking blocks are being proposed on time, explains Pai. 

They’re checking voting rates — making sure validators are able to vote on blocks — and that the voting quality is roughly of the same quality as it was pre-fork. Then they’re checking to see when a whole epoch finalizes. 

“You let off one big sigh after the first finalization.” Within a few hours, “everything’s running normally, and then everyone just forgets; it’s like the chain has always been running like that, just continuing happily,” adds Pai. 

However, Pai says it’s still possible for something to go wrong under the surface. 

“You’re never completely in the clear. This is not just Ethereum. This is any of the blockchains. Like weird stuff happens for a particular hard fork. It could be an edge case that manifests a month later.”

In May 2023 — a month after the Shanghai hard fork — Ethereum developers went scrambling after the network temporarily lost transaction finality for around 25 minutes, then again for over an hour a day later before the blockchain recovered by itself. 

Finality is considered the point at which a transaction becomes irreversible and permanently recorded on the blockchain. 

“I do remember being very scared that something weird was going on,” said Pai. “It’s unintended behavior, and when there’s unintended behavior, you don’t know what else is unintended or what else could be going wrong.”

A post-mortem later pointed to a flaw affecting two consensus clients, Prsym and Teku, which issued a fix not long after — though it still took a toll on developers and others in the Ethereum ecosystem.

Another finality incident happened during the Altair fork on the Ethereum Beacon Chain in 2021 when approximately 25% of blocks went missing, which ended up being due to one large staking operator not having upgraded its infrastructure in time. 

“That’s why we spend a lot of time on due diligence and make sure that things are correct and good,” Lucas Saldanha, lead protocol engineer at Consensys and Ethereum core developer, tells Magazine. 

“But I think you can never be 100% sure with anything in life, really. It’s more about just embracing it and making sure that you’re doing the best that you can.”

In May, the stakes are just as high again, as Ethereum will be going through its biggest upgrade since transforming to a proof-of-stake blockchain via the Merge in 2022.

It’s the biggest fork in terms of the number of features and spec changes, explained Saldanha. 

A whopping 11 code changes, or Ethereum Improvement Proposals, will be activated to make the network cheaper, more efficient and generally more user-friendly. The last major upgrade, Dencun, in March 2024, included nine EIPs.

However, the Pectra upgrade was plagued by issues on the first two testnets. 

In February, the Holesky testnet took two weeks to achieve finality due to a client software bug that happened for a “very specific, almost irritating” reason as a result of Holesky’s testnet setup, according to Pai. 

Then in March, Ethereum’s second testnet for Pectra, Sepolia, encountered a different issue that led to many execution layer clients not including transactions in blocks. That was a problem specific to the configuration of the testnet and should not be able to occur on mainnet.

Nixo tells Magazine that developers are “tired” and are just hoping to move on to Ethereum’s next upgrade, Fusaka, after working on Pectra for the last year and a half. 

“This has been a really long fork for them. People are exhausted.”

She says this is another reason why Ethereum developers have been pushing for more frequent upgrades to the protocol.

In February, during an “All Core Devs” meeting, Ethereum devs and ecosystem leaders agreed to shorten the amount of time between each upgrade. Crypto venture firm Paradigm has also called for Ethereum developers to ship faster than one change per year. 

Nixo says part of this desire comes from developers simply wanting to be able to “reset” more often when they start work on a new fork. 

“Before all of this testnet stuff happened, there was a narrative that, like, we wanted accelerationism; we wanted fork cadences to go faster,” said Nixo. 

“But part of that that wasn’t really talked about was that developers are just tired of working on the same thing for so long. Like, as you go on, if you don’t freeze the spec, more and more things get added.” 

“And honestly, they are just tired of working on something for a year, a year and a half.”

Saldanha explains that with a faster release cadence, developers will be making smaller changes each time, which could make it a little less risky each time there’s an upgrade. 

“It’s like a reset when you start working on a new fork,” added Nixo. 

Logically, though, faster forks could also be riskier. Both Saldanha and Pai say there also needs to be a balance between speed and caution. 

Saldanda says there are limits to how fast Ethereum can ship, given its decentralized nature and the challenges around working with developers from all over the world. 

“In reality, when it comes to Ethereum and the way that we do things, there are a lot of other factors that don’t really allow us to have a super quick development cycle,” he said. 

Pai notes that Ethereum has five consensus clients and five executive clients, meaning that an individual can run any one of 25 different combinations. 

“So, that means while, in principle, everything is written to spec, there are lots of edge cases. So, you really need to be very sure that all 25 combinations, in principle, could work, right? That there isn’t some weird communication error that comes up for some weird case that causes trouble for some set of people.”

“If it was just one monolithic person pushing out on a monolithic client, they could do something faster and continuously integrate.”

On the upside, that makes it much less likely they’ll all fail at once, which is the rationale for having so many clients. 

Pai says he has seen talk of two forks a year but stresses that one can’t go too fast and skip important steps in this process.

The Ethereum mainnet alone holds some $123 billion in stablecoins and holds a 57% market share of real-world tokenized assets. 

“We definitely value that trust and are not going to, you know, muck it up,” says Pai. 

EtherEthereumhard forkPectraupgrade

Read also

Hodler’s Digest

Criminal at Bitcoin 2024, BTC Strategic Reserve Bill, and more: Hodler’s Digest, July 28 – Aug. 3

by
Editorial Staff
10 min
August 3, 2024

Sen. Lummis introduces Bitcoin Strategic Reserve Bill, Bitfinex hacker shows up at Bitcoin 2024, and Trump Bitcoin sneakers: Hodler’s Digest.

Read more

Hodler’s Digest

BTC white paper hidden on macOS, Binance loses AUS license and DOGE news: Hodler’s Digest, April 2-8

by
Editorial Staff
7 min
April 8, 2023

Binance Australia Derivatives’ license revoked, the Bitcoin white paper is hidden on modern macOS and Dogecoin prices spike after Twitter’s icon change.

Read more