On March 28, 2024, Ethereum developers gathered over Zoom for All Core Developers Execution (ACDE) Call #184. Chaired by Ethereum Foundation (EF) Protocol Support Lead Tim Beiko, the ACDE calls are a bi-weekly meeting series where developers discuss and coordinate changes to the execution layer (EL) of Ethereum.

This week, developers shared insights on what was causing an uptick in the number of missed blocks on Wednesday, March 27. Prysm developer Terence Tsao said the uptick was caused by issues in the Bloxroute MEV relay, which the Bloxroute team is in the process of resolving. Developers also discussed the takeaways from new research on Ethereum state and history growth conducted by the Paradigm team. Developers approved the inclusion of two retroactive Ethereum Improvement Proposals (EIPs) in Prague/Electra, EIP 7610 and 7523.

Finally, they shared updates on the development of other candidate-EIPs for the upgrade such as EIP 7547 (inclusion lists), EIP 5920 (PAY opcode), and EIP 7545 (Verkle proof verification precompile).

Missing Block Incident on Mainnet

On Wednesday, March 27, there was an uptick in the number of missing blocks. Usually, 2% to 4% of blocks are missed on Ethereum every 30 minutes. However, this percentage rose to over 14% in a matter of hours during a period when the network was experiencing a high volume of blob transactions. The price of blobs in this period increased more than 10-fold. Tsao said the issue of missing blocks was immediately resolved once the Bloxroute team shut off their MEV relay. The details of what was causing the Bloxroute relay issues are not yet known and the Bloxroute team is working on a fix that they will share, along with a full post-mortem of the issues, in the coming days.

"So, yesterday's missed block [issue] wasn't so much about like clients couldn't handle that type of workload, because basically … all the [missed] blocks were caused by the Bloxroute issue. There's probably like 99% of it but then there is still that fundamental issue of okay what happens under yesterday's traffic, and I suspect, yes, clients may import blocks slower than before, but that's something that I don't really have strong evidence on, that's something that still remains to be seen," said Tsao. In response to the missing blocks incident, the Lighthouse client team has issued a "hot-fix" release improving node performance and stability. Additionally, while investigations are ongoing, CEO of Bloxroute Uri Klarman said on X that he does not believe the issues to be related to the Bloxroute relay but rather the way blobs are propagated in general on Ethereum.

Ethereum Foundation Developer Operations Engineer Parithosh Jayanthi asked whether the incident should cause developer to re-evaluate client circuit breaker conditions, which are conditions that automatically cause validator nodes to fall back to local block production. The default for the circuit breaker condition in most clients is the event of five consecutively missed slots. Tsao noted that circuit breaker conditions that can be triggered too easily are a potential attack vector that sophisticated MEV actors could exploit.

Prysm developer "Potuz" said that in his view the incident highlights a lack of client diversity implementations within relays and lack of communication between relays and protocol developers. "Terence has been talking about these blobs for over a week and no one paid attention to this and once it exploded, it only took a couple of phone calls to get the right relays to actually look at their logs. This is unacceptable," said Potuz.

Some developers recommended creating written posts next time when it comes to reporting network irregularities for higher visibility in the Ethereum ecosystem. For further discussion about the missing block incident, Ethereum Foundation Researcher Alex Stokes encouraged developers to participate in the next MEV-Boost Community Call.

State and History Growth Data Analysis

Storm Slivkoff, a data scientist engineer at Paradigm, presented a new analysis on Ethereum state and history growth. Based on his findings, Slivkoff noted that state growth is not a major bottleneck to Ethereum scalability. "We find that existing consumer hardware can sustain current rates of state growth for a very long time, probably decades. Note that I'm only talking about storage capacity and memory capacity here. This isn't saying anything about the reads or the writes to state under this framing," said Slivkoff, adding that in his view the "silent killer" of Ethereum is history growth.

In the written analysis, the Paradigm research team explains, "State is the set of data necessary for building and validating new Ethereum blocks. State is composed of contract bytecode, contract storage, account balances, and account nonces. History is the set of data necessary for syncing a node from Genesis to the latest block. History is composed of blocks and transactions. State and history are non-overlapping datasets." Slivkoff added that history is growing at a significantly faster rate than Ethereum state. The largest use case blowing up history growth rates are rollups and other types of protocols requiring bridges to Ethereum.

Slivkoff recommended that developers seriously consider expediting EIPs addressing history growth such as EIP 4444 and EIP 7623 in the next Ethereum upgrade, Prague/Electra. He also said that further analysis will be done to analyze other scaling bottlenecks on Ethereum and apply these methodologies to analyzing scaling bottlenecks on rollups as a next step for his team's research. All the data will be open sourced for public consumption and feedback is welcome, said Slivkoff.

After Slivkoff's presentation, developers debated the different ways that history growth could be addressed in the near term. As discussed on ACDE #180, developers are building out robust alternative networks where history data past a certain period, such as history data before the Merge upgrade, could still be accessed by users in the event the data becomes inaccessible through Ethereum nodes. For further discussions on history expiry and alternative options for servicing history data, Geth developer "Lightclient" recommended developers continue the conversation on the Ethereum research and development Discord channel in the sub-channel topic titled "history expiry".

Retroactive EIPs 7610 and 7523

Developers agreed to implement EIP 7610 and 7523. These are retroactive EIPs that would add rules to the Ethereum protocol that can be applied retroactively from the beginning of the network to further constrain certain types of behavior on chain. These EIPs have the benefit of simplifying Ethereum test cases and limiting the scope of various edge cases such as the edge case of the creation of empty accounts. Two EIPs that have already been retroactively applied include EIP 2681 and 3607. Developers agreed to activate two more retroactive EIPs in Prague/Electra. For background on what behaviors these EIPs constrain, refer to prior call notes here.

EIP 2537, BLS Precompiles

The Geth client team has completed a few benchmarks to estimate the gas cost of EIP 2537 BLS curve operations. These new operations will be activated in the Prague/Electra upgrade and developers are currently weighing the pricing of these operations. A representative from the Reth team said that his team will also complete additional benchmarks for BLS curve operations to assist with determining the gas costs for these operations.

EIP 7547, Inclusion Lists

As discussed on ACDC #130, developers are strongly considering EIP 7547 for inclusion in the Prague/Electra upgrade. This week, Ethereum Foundation Researcher Mike Neuder shared an update on how EIP 7547 could be modified to be forward-compatible with account abstraction. Account abstraction is an ongoing initiative to introduce greater flexibility and programmability to externally owned accounts (EOAs), which are user-controlled accounts, on Ethereum. Neuder presented three different ways to address compatibility issues between EIP 7547 and account abstraction EIPs. About these solutions, Neuder said, "This does feel like complications on inclusion design, but I do think there are these three options that actually do work, and I also don't think that there's going to be a silver bullet to fix this…I don't think we're going to find a better design that doesn't have these same issues that it has to resolve."

Beiko recommended continuing the conversation on inclusion list design in a separate breakout meeting for the sake of limited time.

Other Candidate EIPs for Prague/Electra

Next, developers ran through a list of other candidate EIPs for the Prague/Electra upgrade. They included:

• EIP 5920 (PAY opcode): Ethereum Foundation Researcher Sam Wilson noted that testing efforts for this opcode have already started.

• EIP 7609 (Decrease base cost of TLOAD/TSTORE): Contributor to the Vyper compiler Charles Cooper reiterated his view that the TLOAD and TSTORE opcode should be priced more cheaply in the EVM.

• EIP 2935 and 7545 (Save historical block hashes in state and Verkle proof verification precompile): Geth developer Guillaume Ballet presented these two proposals as code changes that would offer future benefits for the Verkle implementation and in the meantime, help alert the broader Ethereum ecosystem about the forthcoming Verkle upgrade.

• Ethereum Object Format (EOF): Besu client maintainer Danno Ferrin said that EOF EIPs are being implemented by multiple client teams and reference tests are being written for them. He asked that developers refer to

  the EOF readiness matrix

  for a more detailed update.

• EIP 7212 and EIP 3074 (Precompile for secp256r1 Curve Support and AUTH/AUTHCALL opcodes): Besu developer Matt Nelson highlighted these two EIPs that are being implemented by Layer-2 rollups. He emphasized that to encourage compatibility between Ethereum and rollups, these two EIPs should be adopted in Prague.

• EIP 7664 (Access Key opcode): OPLabs developer "Protolambda" presented an alternative proposal to EIP 3074 that utilizes access lists to enhance the capabilities of the AUTH/AUTHCALL opcodes.

• EIP 6493 (SSZ transaction signature scheme): Protolambda also expressed his support for SSZ-related code changes to improve the security and efficiency in verifying Ethereum data.

Developers did not have time to discuss which EIPs on this list should be prioritized for Prague. Beiko said that time will be blocked off at the start of the next ACDE call in two weeks to go over this list again. "In the next few weeks, we should dive deeper into all the stuff that's been brought up today and aim to make a decision then. I think the implication is anything that's not fully figured out or specified two weeks from now is probably not going to make it into this fork if we want to move forward," said Beiko.

Public link

Please use the above public link if you want to share this noodl on another website.