Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flate: Improve level 1-3 compression #678

Merged
merged 1 commit into from Sep 25, 2022

Commits on Sep 25, 2022

  1. flate: Improve level 1-3 compression

    Use 5 byte hash instead of 4 byte hash.
    
    This improves compression in most cases and will also yield faster decompression. Little to no performance impact.
    
    Before/after:
    ```
    file	out	level	insize	outsize	millis
    nyc-taxi-data-10M.csv	gzkp	1	3325605752	922273214	14065	225.49
    nyc-taxi-data-10M.csv	gzkp	1	3325605752	846471964	14564	217.76
    
    nyc-taxi-data-10M.csv	gzkp	2	3325605752	883782053	15683	202.22
    nyc-taxi-data-10M.csv	gzkp	2	3325605752	815766227	15057	210.63
    
    nyc-taxi-data-10M.csv	gzkp	3	3325605752	878726683	17308	183.24
    nyc-taxi-data-10M.csv	gzkp	3	3325605752	807241782	17184	184.56
    
    nyc-taxi-data-10M.csv	gzkp	4	3325605752	789447233	20651	153.57
    nyc-taxi-data-10M.csv	gzkp	4	3325605752	789447233	20862	152.02
    
    file	out	level	insize	outsize	millis	mb/s
    enwik9	gzkp	1	1000000000	382781160	5713	166.90
    enwik9	gzkp	1	1000000000	374131553	5926	160.90
    
    enwik9	gzkp	2	1000000000	371351753	6131	155.55
    enwik9	gzkp	2	1000000000	361881529	6007	158.74
    
    enwik9	gzkp	3	1000000000	364881746	6891	138.39
    enwik9	gzkp	3	1000000000	355065173	7043	135.39
    
    enwik9	gzkp	4	1000000000	342732211	8339	114.36
    enwik9	gzkp	4	1000000000	342732211	8327	114.52
    ```
    klauspost committed Sep 25, 2022
    Copy the full SHA
    f46078c View commit details
    Browse the repository at this point in the history