Ankr Contributing to BNB Chain, Implements Erigon Performance Upgrade

This is a community post written by the Ankr team and published on Binance.org Blog.

BNB Chain began life as a fork of go-ethereum. The aim was to bring interoperability and programmability alongside compatibility with existing Ethereum Smart Contracts.  The increasing success of BNB Chain has led to significant scalability issues that trace back to its origins as a fork of Geth.

One of the key issues for running a BNB Chain Archive Node has been demands on local storage. Archive Nodes can store approximately 17TB of data.

This cumbersome storage size requires significantly more time to index or compact the data, and reduces the performance of the Geth client when requesting data. Geth (Go Ethereum) is a command-line interface used to operate in conjunction with the EVM – the Ethereum Virtual Machine.

EVM compatibility is one of the major benefits of the BNB Chain as it supports the extensive existing Ethereum tooling, block time speeds, and low costs. We need to create a more efficient storage mechanism to keep the BNB Chain operating at peak performance levels while ensuring Geth is running smoothly.

Introducing the Erigon Solution

Erigon originally started as turbo-geth and an improvement of Geth. However, it soon evolved into a very different product to Geth hence the new name.
Ankr is implementing an Erigon BNB Chain version based on https://github.com/ledgerwatch/erigon to solve the existing problems in the BNB Chain. The new version will be open-source and available to all. Implementation of the Erigon client aims to improve efficiency in data storage and increase blockchain performance in several key ways:

• Erigon features a modular design enabling parellelized development of the client.
• There are fewer Read/Write operations with the database when interacting with state.
• Preprocessing of data outside the storage engine improves database efficiency significantly.
• Synchronization via a series of steps allows for much faster sync.

Why Erigon?

The improvements in the Erigon client have significant benefits over Geth and other existing ethereum clients. Specifically:

• Drastically reduced disk storage - 1.2TB for Archive Node, 430GB for Pruned Node.
• Faster sync speed > 10 blocks per second
• Fully bootstrapped archive node in <3days.
• Crash resilience - Erigon’s database can withstand power failures.
• Modularity - P2P and web3 RPC services can be run as components.

This solution is mutually beneficial for BNB Chain and Ankr. Creating Erigon as open-source means that BNB Chain node operators can freely use it and be able to contribute their computing power to the Ankr Protocol and multi-chain RPC network. With an incredibly fast and reliable network of BNB Chain nodes, Ankr will be better able to serve the blockchain and dApp developers that depend on the protocol to communicate with the BNB Chain.

Challenges and Improvements With Ankr’s Implementation

Erigon is an amazing solution for the above reasons. However, it is not fully compatible with BNB Chain since the BNB Chain has a unique way of handling consensus and block processing. For that reason, Ankr has been working with the BNB Chain team to merge Erigon capabilities to the BNB Chain and resolve some key migration challenges. A key priority has been to ensure compatibility with all nodes throughout the migration from BNB Chain Geth to Erigon,

Features that were migrated from BNB Chain node:

• Genesis blocks and chain configurations - Erigon stores all genesis information inside sources itself instead of JSON files and doesn’t require pre-initializing the blockchain state before running the node. This is useful but requires additional support for all possible geneses in source files. It also helps to run nodes by providing a chain flag with BNB Chain value.
• Parlia consensus engine - BNB Chain uses its own consensus engine called Parlia whilst Erigon’s consensus interfaces are not fully compatible with Geth’s. We therefore decided to modify some interfaces to make them compatible with both systems without sacrificing compatibility with block processing. This makes it much easier to merge new Erigon changes.
• BNB Chain’s compatible system calls - Erigon already supports system calls but they’re not designed to be compatible with BNB Chain’s system calls due to differences in callee and gas limits. The key difference of BNB Chain is that every system transaction requires an intermediary block finalization that is not supported by Erigon. Additionally, this can lead to some gas calculation issues if multiple system calls modify the same state objects.
• Forks - We support all BNB Chain forks, including Ramanujan, Niels, MirrorSync, and Bruno.
• Block processing - BNB Chain uses a somewhat tricky block processing method that appends system transactions and receipts onto every block execution. Due to block modification, it also changes transactions, receipts root, and state root. That is why the block processing part of Erigon was also changed because the terms of the Erigon consensus engine cannot modify the state; it can only verify it.
• System contracts - Of course, all BNB Chain system contracts are supported with a transport layer that facilitates BNB cross-communication functionality.

Key Milestones

In the first release of Erigon for BNB Chain, we decided not to enable Validator Mode and instead let it work just as a full blockchain node. Currently, Erigon doesn’t support accounting to avoid creating disruptions to the Erigon node. We will support this function correctly with the Erigon team.

The first milestone will be to ensure that the Erigon solution works without errors in implementation with BNB Chain full nodes (both Mainnet and Testnet).

Once full node capabilities are complete, we will verify that Erigon is compatible with the BNB Chain validator nodes.

Erigon’s Improvements on BNB Chain

• Increased Speed & Performance - Erigon implements a much faster MDBX storage library (instead of LevelDB) that can reduce database size and improve BNB Chain performance. MDBX is one of the fastest embedded databases and the most lightweight. Of course, MDBX has some limitations. For example, it supports only 2^31 filesystem pages to be addressed, and it limits database size to 8 TB, but it can be fixed by increasing the page size on FS. Currently, it takes around 2.5TB to store a full archive BNB Chain node in a database using Erigon and an MDBX driver. MDBX drivers bring performance up by 30% compared to other databases like RocksDB or LevelDB.
• New RPC Daemon Capabilities - One of the key features of Erigon is the RPC daemon. By removing all RPC functionality from the node, we can create an independent transport function between the node and RPC daemons to let developers scale their RPC using remote database functionality. This is a very powerful mechanism that makes it possible to build clusters from the node by only setting one Erigon full sync mode and increasing its scalability.
• Faster Synchronization Process - Syncing nodes can be a long and arduous process, but Erigon speeds this along with stages that batch data together more efficiently and minimize database overwrites. The key idea of staged sync is to split the synchronization process into 15 parts, like downloading headers or bodies, block execution, or state verification. Such division allows the utilization of all resources of the machine by using batch processing and shared caches. All intermediary caches are dropped after each phase which saves disk space and IO operations. Most of the stages allow specific optimizations, which are hard to implement without a staged sync architecture. The heaviest stage is still block execution. This is hard to parallelize due to the strict sequential nature of transactions themselves. However, the impact of all other parts on the sync process is minimized.

Benchmark Testing and Results

Upon benchmark testing, the Erigon BNB Chain version performed remarkably well. Here is a brief overview of our process.

Hardware Specs

We deliberately used a below-average server from Digital Ocean to show the excellent performance of Erigon for BNB Chain. The server used included:

• 4 vCPU
• 8 GB memory
• 160 GB disk for OS
• 1T GB disk for BNB Chain testnet archive data; digital

Synchronization Process

The Testnet synchronization process started on December 6 at 21:34:41 and lasted to the latest height at December 7, 22:54:31. Completed on block 14778509.

The entire process took 25 hours and 10 minutes. However, Erigon for BNB Chain is using only one processor from the CPU, which leaves a lot of room for future improvement. Having only 1 CPU makes it very challenging to speed up because blocks & transactions executions are inherently consecutive, not parallel. At Ankr, our preliminary estimate of this code refractory could be approximately 4-5 times faster than it is now although it is very challenging to implement. But again, this is much better than the current Go-Ethereum based clients already.

Benchmark

The Ankr team used ethspam and versus to test the performance of the Erigon client. Conducting tests with 100, 200, 300 concurrent requests, respectively, we encountered unquestionably outperforming results with 0% error. We compared this to an identical server using the Geth Goreli testnet with 100 requests, and the test encountered 6.75% connection errors.

Erigon with 100 concurrent requests

Erigon with 200 concurrent requests

Erigon with 300 concurrent requests

GETH Goreli testnet with 100 concurrent requests

We choose a Digital Ocean server with the exact same specifications for similar benchmark testing. The results show much worse performance (in terms of tp95 tp99) with 6.75% connection errors.

Future Work

There are even more opportunities for improvement that Ankr can follow now. And we will be in conversations with the BNB Chain team to coordinate and plan our efforts. However, here are some potential priorities:

• Keep improving the performance of Erigon for BNB Chain by utilizing multi-core of CPU
• Enable validator mode
• BNB Chain 2.0

Follow our GitHub to learn more about Erigon and follow future test results.

Benefits for the BNB Chain Ecosystem

Ankr has a close relationship with the BNB Chain ecosystem, and we are proud to announce this new project that will benefit the BNB Chain community and applications.

Ankr offers a further boost to BNB Chain developer capabilities with our bscrpc.com public RPC. This open access, free RPC allows anyone to make request calls to the chain and receive responses exactly as if they were running a BNB Chain full node themselves.

“The BNB Chain is seeing an all-time high in transactions as emerging markets heavily adopt it for DeFi, NFT’s, and gaming. The low gas fees and EVM compatibility make it a favorite. However, this comes at a cost: nodes are struggling to keep up. With Ankr open-sourcing our custom implementation of Erigon, we can help node providers keep at block height, ensuring the BNB Chain community keeps thriving.” – Josh Neuroth, Ankr’s Head of Product

“We are proud that BNB Chain is reaching lofty goals as an open protocol and community-driven ecosystem.” – Samy, BNB Chain’s Ecosystem Coordinator

BNB Chain Core team remark: We believe that all participants in the BNB Chain ecosystem will greatly benefit from these network upgrades. This is the evolution of BNB Chain to become faster and more efficient, enabling it to handle the ever-growing demand and staggering number of transactions. BNB Chain continues its journey towards mass crypto adoption and it will continue to evolve whenever new challenges arise.

Follow us to stay updated on everything BNB Chain!

Website | Twitter | Telegram | Youtube | Gitcoin