Poc: test read / write performance of chunked 32-bit roaringbitmap

[xin-hedera]

Problem

One idea to increase roaringbitmap compression ratio is create a global contiguous transaction index, and use the roaringbitmap to mark the transaction index instead of sparse, rarely contiguous consensus timestamp. In doing so, there is a much higher change to encode the bits into run-length containers instead of the other two types which don't save as much space.

Also we want to use 32-bit roaringbitmap to reduce both write and read amplification. This however requires the table to have three columns, entity_id, high_value, and bitmap, where the high value is the high 32-bit of the transaction index.

The design would change how the rows are created, merged, and optimized during ingestion, as well as how to query the data based on the rest api query filters since essentially the bitmaps are not split into chunks.

The goal of the ticket is to run PoCs, given transactions with production env pattern, test and identify the potential bottlenecks, and optimize if possible.

Solution

check the description

Alternatives

No response