feat: add bloom filter when write sst

[jiacai2050]

Which issue does this PR close?

Part of #363

Rationale for this change

Described in #363, bloom filter is beneficial for columns with high cardinality, so this PR append bloom filter to sst's meta when write SST files.

What changes are included in this PR?

Update sst write process, loop record batch twice:

• first to build bloom filter
• second to write to underlying storage.

Note: for now, all columns is appended with bloom filter, some columns may not suitable for it, we can remove them in future to reduce meta size.

Are there any user-facing changes?

No

How does this change test

Add UT: test_partition_record_batch

After we support prune by bloom filter, we can add more test cases.

[tustvold]

FWIW parquet has a form of bloom filter support, and I, and likely others, would be very interested in collaborating to get support added to arrow-rs/DataFusion if this was something you were amenable to. Just thought I'd mention it, as it has come up in a few IOx discussions lately 😄

[alamb]

This is very cool work @jiacai2050 👍

Along with @tustvold I would love to help collaborate on getting Bloom filter support into parquet (and then also datafusion) -- my first contribution to the process is writing up a ticket apache/arrow-rs#3023

[jiacai2050]

@tustvold @alamb Thanks for your tips. Great write-up.

The bloom filter support in CeresDB is in early stage, we have just drafted an initial design how to add bloom filter, which is different from what parquet-format said.

Simply put, bloom filter is row-group based, and we intend to encode them in key_value_metadata instead of in each row group's meta, so we can read them out in one IO.

We will verify this design in our case ASAP, if it's works as expected, we would be happy to contribute it to parquet crates.

[alamb]

Simply put, bloom filter is row-group based, and we intend to encode them in key_value_metadata instead of in each row group's meta, so we can read them out in one IO.

Sounds like a good plan -- thank you @jiacai2050 ! I think using a custom format in the metadata sounds like a good idea for CeresDB initially (it will be much faster than working it through the rest of the ecosystem). We would love to hear about your experience in implementing it

[ShiKaiWi]

LGTM

[alamb]

🎉