Ethereum: Understanding CompactSize and VarInt Encoding
The Ethereum blockchain has long been plagued by issues related to data compression and encoding. Two of the most widely used encodings in the ecosystem are CompactSize and VarInt (Variable Int), which have been referenced interchangeably until recently. In this article, we will delve into the differences between these two encoding schemes and explore why they were previously confused.
CompactSize Encoding
Pieter Wuille’s definition of CompactSize encoding is a crucial point to establish clarity. According to his paper “On the compactness of Ethereum transactions,” CompactSize encoding is defined as a simple replacement of certain characters in the transaction data with shorter codes, ultimately reducing the size of the data while maintaining its essential information. This approach aims to minimize the storage requirements for the transaction data without sacrificing its security or integrity.
In contrast, Greg Walker’s definition of VarInt encoding emphasizes the use of an array of variable-length integers (VLIs) to store and transmit data. These VLIs are used to represent complex data structures in a compact and efficient manner. VarInt is often seen as a more sophisticated encoding scheme compared to CompactSize, but both encodings can be used to reduce the size of transaction data.
VarInt Encoding
VarInt encoding has been widely adopted in many blockchain networks, including Ethereum. According to the Bitcoin Wiki, VarInt is defined as an array of unsigned integers that represents a data structure. The purpose of VarInt is to provide a compact and efficient way to store and transmit large amounts of data on the network.
The key differences between VarInt encoding and CompactSize encoding lie in their approach:
- Structure: VarInt uses an array of VLIs, while CompactSize replaces certain characters with shorter codes.
- Purpose: VarInt is designed for storing and transmitting complex data structures, whereas CompactSize is primarily used to reduce the size of transaction data.
Why the confusion?
It’s not surprising that the Bitcoin Wiki initially claimed that CompactSize is not related to VarInt. The reason lies in the fact that Pieter Wuille’s definition explicitly describes CompactSize as a “simple replacement” approach, whereas Greg Walker’s definition emphasizes the use of VLIs and complex data structures.
In reality, both encoding schemes can be used together or independently. While CompactSize can be used to reduce transaction data sizes, VarInt is still necessary for storing large amounts of complex data, such as smart contract code or network configurations.
Conclusion
The differences between CompactSize and VarInt encoding are clear once the correct definitions are understood. While both encodings aim to improve data compression, their approaches differ significantly. Pieter Wuille’s definition of CompactSize highlights its simplicity and focus on transaction data reduction, whereas Greg Walker’s definition emphasizes the use of VLIs for complex data storage.
By understanding these differences, developers and users can choose the encoding scheme that best suits their specific needs, ensuring efficient data transmission and storage on the Ethereum blockchain.