BNB Smart Chain Annual Storage Report 2024

Introduction

In 2023, the BNB Smart Chain (BSC) maintained consistent traffic volumes, witnessing a notable increase in market activities due to inscriptions in December. These developments over the past year have significantly influenced BSC's storage demands. In this report, we will learn:

1. How do the storage statistics differ from the previous year?
2. What phenomena cause the storage difference?
3. The challenges faced and the proposed direction to resolve them

Storage Overview

All storage statistics are obtained by setting up a full node with Path-based Storage Scheme (PBSS) and PebbleDB synced to block 34840595, and were generated on 31st December 2023.

The following table shows an overview of the storage result:

The following visualization shows the storage distribution of each major component:

As shown, block data takes up the majority of the storage, followed by the world state and metadata. By comparing with the storage layout in December 2022, which was announced in BNB Smart Chain Annual Storage Report 2023, the summary is as follows:

• The total storage size increased from 1.73TB (correction with ~130GB transaction index) to 2.45TB, a growth rate of 41.6%.
• The storage capacity of each major storage component is shown below, and the growth rates are 42.6%, 42.5%, 42.9%, and 34.4% respectively.

Block Data

The following graph shows the year-over-year block data comparison:

In 2023, BSC saw a notable increase in its data storage requirements, particularly in block body sizes, which expanded by 256GB, marking a 46.4% growth rate. Additional components such as receipts, headers, and codes also experienced significant increases, growing by 185GB, 6.68GB, and 4.73GB respectively, with their growth rates standing at 37.95%, 49.1%, and 30.5%. This expansion pace represents a slowdown compared to 2022, attributed to the reduced transaction per second (TPS) in a bear market.

The substantial block size presents several challenges. One key issue is the necessity to store all blocks from the Genesis block to the most recent, consuming extensive disk space that will only continue to grow. However, executing the most recent blocks does not require access to historical block data. This situation presents an opportunity to explore optimization techniques that could potentially reduce the storage needs of a node by excluding this historical data.

Furthermore, the size of each block increases with higher transaction throughput. From the average block size and daily transaction number charts on BscScan, the average block size is around 40k-50k and the average TPS is around 44.  In December, the block size once reached 250k and the TPS reached more than 1k, which is consistent with the popularity of the entire crypto market. Higher TPS means larger block data size, which demands more disk bandwidth and larger disk space.

Exploring the database mechanics further, initially, recent blocks are stored in a key-value (KV) database. When these blocks age beyond a certain point, termed the ancient threshold, they are transferred to the ancient database. This transfer process, unfortunately, results in some disk bandwidth inefficiency. Additionally, it's important to note the implications of EIP-4844. With the adoption of EIP-4844 by the BSC, an increase in block size is anticipated due to the incorporation of blobs. Although the storage required for blobs may not expand over time, it will nonetheless impose an additional demand for disk space on the part of node operators.

World State

Trie

From the table above, we see a huge surge in the number of accounts, particularly for contract accounts which increased by 121.4%. This indicates a healthy growth and activity level within the BNB Chain ecosystem even during the bear market. However, this also leads to an increase in trie storage size with a growth rate of 42.6%. 

Diving deeper into the MPT composition, the following diagram shows the proportion of trie nodes on each trie level:

The deeper the nodes are in the trie, the longer the reading latency,  which may impact the node performance. Most trie nodes are concentrated in the 7th and 8th levels of the trie, which is still considered normal.

Snapshot

Snapshot is a flat key-value representation of the trie. Hence, the increase in the number of accounts in the trie would also increase the account snapshot size.

Big contract accounts

The unbounded nature of contract size allows for a single contract to potentially grow as large, or even larger, than the entire account trie. In light of this, an analysis was conducted on "big contract accounts," characterized by their extensive storage sizes, manifested through substantial KV pair volumes written by the contract.

These contracts, with their significant storage demands and complex, multi-layered MPT structures, could lead to storage amplification issues, adversely affecting node performance. Presented below is a table detailing the number and proportion of trie nodes for the top 20 contracts:

Since the database only stores the hash of the account address, it is not easy to obtain the original account address directly. We attempted to identify the original addresses of these large players and have listed the top 5 below:

Future Development

Blockchain is highly IO-bounded. Higher transaction throughput means more disk bandwidth, and larger database size also affects the database performance and the overall system performance. 

Reasonable data storage solutions and utilization of the disk bandwidth are the keys to effectively improving overall system throughput. Below are some proposals and directions we can research based on the analysis that we’ve done on this storage report:

1. Separated databases for block data and state dataBlock data is stored sequentially while state data is stored randomly in the database. Split database by data pattern will make disk bandwidth usage more reasonable and improve the whole performance.
2. Segmented History Data MaintenanceIt can help resolve the problem of increasing history block storage on the BSC for validators and full nodes. They only need to maintain a limited range of blocks. 
3. State expiry in contract level to reduce current world state sizeThe current world state data size is continuously increasing, which will impact the network's performance. We need to build some strategy to keep it under control.  Some storage tries may be rarely or no longer used. These storage tries’ state data can expire to reduce the whole state size.
4. Build a high-performance state database

Currently, the state data is constructed on MPT and stored in a generic store such as LevelDB. The index performance is not good enough and our team is working to find a new solution to solve it.

5. Integrate state snapshot into trie database

The state snapshot is used to improve the execution performance and its persistent data overlaps with the trie database. Besides, both state snapshot and trie database have similar complicated and nasty recovery mechanisms to ensure recoverability after panic. So it's beneficial enough to integrate state snapshot into trie database for better robustness and simplicity.

6. Improve the performance of storage tries with huge KV pairsA storage trie with huge KV pairs will make too many levels of MPT, which will impact the access performance. 

Looking Forward

In 2023, BSC implemented the PBSS and PebbleDB to enhance the efficiency of blockchain state storage. As we move into 2024, the continuous and rapid growth of blockchain data presents a significant challenge for maintaining BSC's performance. It is crucial for all stakeholders to collaborate in seeking innovative solutions to enhance BSC's efficiency and cost-effectiveness. Together, let's commit to making BSC more robust and sustainable.