Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite valid keys by ingesting SST files during GC #150

Open
DorianZheng opened this issue Mar 21, 2020 · 1 comment
Open

Rewrite valid keys by ingesting SST files during GC #150

DorianZheng opened this issue Mar 21, 2020 · 1 comment

Comments

@DorianZheng
Copy link
Contributor

Since the write callback is removed by #121 , it's possible to generate SST files of valid keys and to ingest these SST files by IngestExternalFile

@DorianZheng DorianZheng changed the title Rewrite index by ingesting SST files Rewrite valid keys by ingesting SST files during GC Mar 21, 2020
@Connor1996
Copy link
Member

The benchmark of #121 shows Ingest may hurt read performance, see

dbbench (./titandb_bench --key_size="16" --value_size="256" --cache_size="4294967296" --titan_min_blob_size="0" --num="10000000" --numdistinct="10000" --benchmarks="fillseq,overwrite,readrandom" --threads="6" --titan_max_background_gc="6"):
gc_rewrite_mode = kDefault
fillseq: 99.3 MB/s; overwrite: 23.2 MB/s; readrandom: 130.5 MB/s
gc_rewrite_mode = kMerge
fillseq: 102.5 MB/s; overwrite: 59.3 MB/s; readrandom: 122.8 MB/s
gc_rewrite_mode = kFastIngest
fillseq: 101.1 MB/s; overwrite: 61.8 MB/s; readrandom: 82.4 MB/s

go-ycsb (max-background-gc=4, threads=256, readcount=1kw, operationcount=2kw)
gc_rewrite_mode = kDefault
update ops 5w -> 4.53w, read ops 12w
gc_rewrite_mode = kMerge
update ops 5w -> 4.58w, read ops 11w
gc_rewrite_mode = kFastIngest
update ops 5w -> 4.7w, read ops 7w

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants