This week’s upgrade to the MegaETH testnet fixed an elusive performance bug that had caused miniblock time to continuously increase between sequencer reboots. Here’s the story. It’s a story about our philosophy – measure, then build. If one visited MegaETH’s performance dashboard recently, one might see that the miniblock time had been increasing during the week leading up to June 3. Actually, such a trend would start right after every sequencer reboot since the launch of the public testnet. Previously, the frequent upgrades to the sequencer meant that the miniblock time would not increase by any perceivable amount before the upward trend was reset. However, recent upgrades had not required sequencer reboots, and the trend continued for weeks. On June 3, miniblock time almost reached 100ms. With sequencer reboots becoming even less likely in the future thanks to hot backups, it’s time to eliminate the bug once and for all. Since we routinely collect lots of telemetry data for the testnet, the team quickly started digging. The first discovery was that the increase in miniblock time accelerated over time – not only did miniblock time increase, it increased faster and faster. Usually, such symptom would imply that the work involved in building each miniblock increased superlinearly as more miniblocks were built. However, we shot down the hypothesis after some measuring and calculation. We built the miniblock pipeline to be almost fully asynchronous to the EVM so as to achieve arbitrarily low miniblock time. This means that however much time it takes to build a miniblock, the EVM will be executing transactions during the whole time. Thus, the longer miniblock building time would lead to a higher number of transactions per miniblock, but we did not observe that. So, the issue cannot be with building miniblocks. Careful examination of the code confirms this conclusion – no part in the miniblock building process has superlinear complexity. The team expanded the search, and the real culprit quickly surfaced. The time it took to commit EVM blocks had been increasing; further, the committing time was perfectly linear to the number of EVM blocks produced since the last reboot. When committing EVM blocks, the EVM environment such as block height is updated, so the EVM must pause and cannot execute transactions, which means no miniblocks either. There is a fixed 1-second interval between EVM blocks. Within the 1-second budget, a linearly increasing committing time lead to a linearly decreasing duration to run transactions, which in turn lead to a linearly decreasing number of miniblocks produced. If we take its reciprocal, we get the average miniblock time, which is inversely proportional in time. It’s exactly the function shape we saw on the performance dashboard. The math checked out. At that point, we knew exactly what to look for: some procedure whose workload increases linearly over time in the particular part of the code that handles committing EVM blocks. The rest of the work was straightforward. The team pushed the upgrade this week and miniblock time has not crept up. So, what was the lesson? I think it showed again the power when engineering is guided by careful measurements and first principles. The team is working on the other upgrades with the same philosophy. Stay tuned!