Implement multiple Backing Store using Generics

[KhaledEmaraDev]

This PR adds different Backing Database Stores for the Index Structures. This is done using generics. Each Backing Store is behind a feature flag and its dependencies only compile when this features is enabled.

[KhaledEmaraDev]

I had to change the toolchain to version 1.61.0 because of redb.

[KhaledEmaraDev]

@romanz Also, Cargo.toml feature dep: needs cargo 1.56.0 at least. That's why it's failing the CI.

[casey]

@cberner will definitely be interested in this!

[KhaledEmaraDev]

This is still incomplete as for instance in redb. I haven't been able to get a prefix iterator so far. For instance in the test:

let items: &[&[u8]] = &[
    b"ab",
    b"abcdefgh",
    b"abcdefghj",
    b"abcdefghjk",
    b"abcdefghxyz",
    b"abcdefgi",
    b"b",
    b"c",
];

It returns everything after abcdefgh. So it's insert time order not prefix search.

I have been trying to track the prefix manually, but so far have been unsuccessful. It's very slow. However; I'm currently trying a new way of tracking prefixs.

[Kixunil]

I had to change the toolchain to version 1.61.0

Unfortunately, we want to support Debian stable which only has 1.48. I personally wouldn't mind having higher MSRV with opt-in feature but it needs CI modified to use 1.48 to test without it and use 1.61 with it.

[romanz]

Many thanks for the contribution!
Unfortunately, I am not available now - will review in two weeks.

[romanz]

• Please rebase over latest master.
• Make sure the build & tests pass at https://github.com/romanz/electrs/actions/runs/2933929796.
• Could you please publish mainnet indexing performance (e.g. Idea: publish electrs index/query benchmarks #567 (comment))?

[KhaledEmaraDev]

@romanz I rebased on master.

The CI is failing because:

1. Audit: Because the GitHub action is running from a fork.
2. Build: Because the redb backend has one slight problem just like I said above. It's making prefix scans wrong. I'm working on fixing it.
3. Integration: Due to the Rust toolchain. I'll fix it.

Can I get the testnet indexing performance instead? I don't have the space right now to download mainnet.
I can try in a few days, though.

[antonilol]

is this PR still being worked on? i think it would be useful to allow for different databases (maybe helps to optimize for hdd/ssd performance), but instead of choosing one at compile time, a command line flag (choose at runtime) would be better (if you use a precompiled release you can still choose) and use feature flags to choose which database backend get compiled in, especially useful for big dependencies or ones that bind to C code

[KhaledEmaraDev]

@antonilol I think I can make to that backed-in storage backends are selectable using a CLI flag.

I think we should do this after merging this PR first. I still need to enhance the redb logic. I'll try to work on that sometime soon.

[antonilol]

how is this pr going?

are you still working on it, and which things are still missing/planned (like the ones you named here, but maybe more)?

[romanz]

Closing due to inactivity.