s2: Add LZ4 block converter #748
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Single threaded performance
Speed excluding LZ4 encoding:
```
BenchmarkLZ4Converter_ConvertBlock/html-32 22450 49962 ns/op 2049.58 MB/s 559.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32 1933 608889 ns/op 1153.06 MB/s -3943 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32 428572 2816 ns/op 43715.74 MB/s 482.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32 413655 2793 ns/op 44075.51 MB/s 482.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32 163200 7226 ns/op 14171.69 MB/s 136.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32 5451 214569 ns/op 1908.94 MB/s 1840 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32 5100 233525 ns/op 651.27 MB/s 106.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32 5997 205362 ns/op 609.55 MB/s -1427 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32 1948 616892 ns/op 691.78 MB/s 384.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32 1419 835555 ns/op 576.70 MB/s -9125 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32 29461 40528 ns/op 2926.08 MB/s 1.000 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32 5454 214264 ns/op 860.25 MB/s 9303 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32 5216 233042 ns/op 652.62 MB/s 106.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32 5000 232401 ns/op 654.43 MB/s 106.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32 4999 232989 ns/op 652.77 MB/s 106.0 b_saved 0 B/op 0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32 4999 233068 ns/op 652.55 MB/s 106.0 b_saved 0 B/op 0 allocs/op
```
Reference comnpression speed:
```
BenchmarkCompressBlockReference/html/default-32 14392 82754 ns/op 1237.41 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/html/better-32 6663 178830 ns/op 572.61 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/urls/default-32 1174 1002967 ns/op 700.01 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/urls/better-32 607 1942559 ns/op 361.42 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32 196694 6023 ns/op 20436.73 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32 51148 23613 ns/op 5212.85 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32 197408 6015 ns/op 20464.42 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32 51267 23509 ns/op 5236.04 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32 91663 12891 ns/op 7943.73 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32 8433 128542 ns/op 796.62 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/html4/default-32 10000 100385 ns/op 4080.29 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/html4/better-32 5712 204505 ns/op 2002.88 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32 3427 340552 ns/op 446.60 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32 2032 576497 ns/op 263.82 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32 4359 266102 ns/op 470.42 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32 2306 509822 ns/op 245.53 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32 1219 971355 ns/op 439.34 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32 748 1598189 ns/op 267.02 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32 1056 1129044 ns/op 426.79 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32 579 2064813 ns/op 233.37 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/pb/default-32 17960 65621 ns/op 1807.16 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/pb/better-32 7062 159908 ns/op 741.60 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32 3870 302586 ns/op 609.15 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32 2396 495081 ns/op 372.30 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32 3426 342366 ns/op 444.23 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32 2066 581844 ns/op 261.39 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32 3524 340442 ns/op 446.74 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32 1996 577981 ns/op 263.14 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32 3423 340195 ns/op 447.06 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32 2013 578202 ns/op 263.04 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32 3518 340101 ns/op 447.19 MB/s 0 B/op 0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32 2031 581532 ns/op 261.53 MB/s 0 B/op 0 allocs/op
```
Size comparisons (using Go lz4 encoder):
```
=== RUN TestLZ4Converter_ConvertBlock
=== RUN TestLZ4Converter_ConvertBlock/html
lz4convert_test.go:34: input size: 102400
lz4convert_test.go:35: lz4 size: 21195
lz4convert_test.go:52: lz4->s2 size: 20636
lz4convert_test.go:64: s2 (better) size: 18969
lz4convert_test.go:66: lz4 -> s2 bytes saved: 559
lz4convert_test.go:67: data -> s2 (better) bytes saved: 2226
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1667
=== RUN TestLZ4Converter_ConvertBlock/urls
lz4convert_test.go:34: input size: 702087
lz4convert_test.go:35: lz4 size: 292514
lz4convert_test.go:52: lz4->s2 size: 296457
lz4convert_test.go:64: s2 (better) size: 248076
lz4convert_test.go:66: lz4 -> s2 bytes saved: -3943
lz4convert_test.go:67: data -> s2 (better) bytes saved: 44438
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 48381
=== RUN TestLZ4Converter_ConvertBlock/jpg
lz4convert_test.go:34: input size: 123093
lz4convert_test.go:35: lz4 size: 123522
lz4convert_test.go:52: lz4->s2 size: 123040
lz4convert_test.go:64: s2 (better) size: 123097
lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN TestLZ4Converter_ConvertBlock/jpg_200b
lz4convert_test.go:34: input size: 123093
lz4convert_test.go:35: lz4 size: 123522
lz4convert_test.go:52: lz4->s2 size: 123040
lz4convert_test.go:64: s2 (better) size: 123097
lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN TestLZ4Converter_ConvertBlock/pdf
lz4convert_test.go:34: input size: 102400
lz4convert_test.go:35: lz4 size: 83152
lz4convert_test.go:52: lz4->s2 size: 83016
lz4convert_test.go:64: s2 (better) size: 82884
lz4convert_test.go:66: lz4 -> s2 bytes saved: 136
lz4convert_test.go:67: data -> s2 (better) bytes saved: 268
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 132
=== RUN TestLZ4Converter_ConvertBlock/html4
lz4convert_test.go:34: input size: 409600
lz4convert_test.go:35: lz4 size: 81908
lz4convert_test.go:52: lz4->s2 size: 80068
lz4convert_test.go:64: s2 (better) size: 18979
lz4convert_test.go:66: lz4 -> s2 bytes saved: 1840
lz4convert_test.go:67: data -> s2 (better) bytes saved: 62929
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 61089
=== RUN TestLZ4Converter_ConvertBlock/txt1
lz4convert_test.go:34: input size: 152089
lz4convert_test.go:35: lz4 size: 79672
lz4convert_test.go:52: lz4->s2 size: 79566
lz4convert_test.go:64: s2 (better) size: 71608
lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN TestLZ4Converter_ConvertBlock/txt2
lz4convert_test.go:34: input size: 125179
lz4convert_test.go:35: lz4 size: 70801
lz4convert_test.go:52: lz4->s2 size: 72228
lz4convert_test.go:64: s2 (better) size: 65938
lz4convert_test.go:66: lz4 -> s2 bytes saved: -1427
lz4convert_test.go:67: data -> s2 (better) bytes saved: 4863
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 6290
=== RUN TestLZ4Converter_ConvertBlock/txt3
lz4convert_test.go:34: input size: 426754
lz4convert_test.go:35: lz4 size: 207038
lz4convert_test.go:52: lz4->s2 size: 206654
lz4convert_test.go:64: s2 (better) size: 184936
lz4convert_test.go:66: lz4 -> s2 bytes saved: 384
lz4convert_test.go:67: data -> s2 (better) bytes saved: 22102
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21718
=== RUN TestLZ4Converter_ConvertBlock/txt4
lz4convert_test.go:34: input size: 481861
lz4convert_test.go:35: lz4 size: 277731
lz4convert_test.go:52: lz4->s2 size: 286856
lz4convert_test.go:64: s2 (better) size: 264987
lz4convert_test.go:66: lz4 -> s2 bytes saved: -9125
lz4convert_test.go:67: data -> s2 (better) bytes saved: 12744
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21869
=== RUN TestLZ4Converter_ConvertBlock/pb
lz4convert_test.go:34: input size: 118588
lz4convert_test.go:35: lz4 size: 19003
lz4convert_test.go:52: lz4->s2 size: 19002
lz4convert_test.go:64: s2 (better) size: 17686
lz4convert_test.go:66: lz4 -> s2 bytes saved: 1
lz4convert_test.go:67: data -> s2 (better) bytes saved: 1317
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1316
=== RUN TestLZ4Converter_ConvertBlock/gaviota
lz4convert_test.go:34: input size: 184320
lz4convert_test.go:35: lz4 size: 71749
lz4convert_test.go:52: lz4->s2 size: 62446
lz4convert_test.go:64: s2 (better) size: 55395
lz4convert_test.go:66: lz4 -> s2 bytes saved: 9303
lz4convert_test.go:67: data -> s2 (better) bytes saved: 16354
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7051
=== RUN TestLZ4Converter_ConvertBlock/txt1_128b
lz4convert_test.go:34: input size: 152089
lz4convert_test.go:35: lz4 size: 79672
lz4convert_test.go:52: lz4->s2 size: 79566
lz4convert_test.go:64: s2 (better) size: 71608
lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN TestLZ4Converter_ConvertBlock/txt1_1000b
lz4convert_test.go:34: input size: 152089
lz4convert_test.go:35: lz4 size: 79672
lz4convert_test.go:52: lz4->s2 size: 79566
lz4convert_test.go:64: s2 (better) size: 71608
lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN TestLZ4Converter_ConvertBlock/txt1_10000b
lz4convert_test.go:34: input size: 152089
lz4convert_test.go:35: lz4 size: 79672
lz4convert_test.go:52: lz4->s2 size: 79566
lz4convert_test.go:64: s2 (better) size: 71608
lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN TestLZ4Converter_ConvertBlock/txt1_20000b
lz4convert_test.go:34: input size: 152089
lz4convert_test.go:35: lz4 size: 79672
lz4convert_test.go:52: lz4->s2 size: 79566
lz4convert_test.go:64: s2 (better) size: 71608
lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
```
klauspost
force-pushed
the
add-lz4-block-converter
branch
from
8ffba47
to
0751108
Compare
klauspost
force-pushed
the
add-lz4-block-converter
branch
from
207c000
to
95a0452
Compare
kodiakhq bot
pushed a commit
to cloudquery/filetypes
that referenced
this pull request
Mar 1, 2023
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | minor | `v1.15.11` -> `v1.16.0` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>klauspost/compress</summary> ### [`v1.16.0`](https://togithub.com/klauspost/compress/releases/tag/v1.16.0) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.15...v1.16.0) #### What's Changed - s2: Add Dictionary support by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#685 - s2: Add Compression Size Estimate by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#752 - s2: Add support for custom stream encoder by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#755 - s2: Add LZ4 block converter by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#748 - s2: Support io.ReaderAt in ReadSeeker by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#747 - s2c/s2sx: Use concurrent decoding by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#746 - tests: Upgrade to Go 1.20 by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#749 - Update all (command) dependencies by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#758 **Full Changelog**: klauspost/compress@v1.15.15...v1.16.0 ### [`v1.15.15`](https://togithub.com/klauspost/compress/releases/tag/v1.15.15) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.14...v1.15.15) #### What's Changed - zstd: Add delta encoding support by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#728 - huff0: Reduce bounds checking by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#734 - huff0: Assembler improvements by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#736 - deflate: Improve level 7-9 by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#739 - gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by [@​willbicks](https://togithub.com/willbicks) in [klauspost/compress#740 - zstd: Don't allocate dataStorage when using byteBuf by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#741 - huff0: Speed up compression of short blocks by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#744 - zstd: Handle dicts by pointer, always by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#743 - fse: Optimize compression by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#745 - Retract v1.14.1-v.1.14.3 by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#742 #### New Contributors - [@​willbicks](https://togithub.com/willbicks) made their first contribution in [klauspost/compress#740 **Full Changelog**: klauspost/compress@v1.15.14...v1.15.15 ### [`v1.15.14`](https://togithub.com/klauspost/compress/releases/tag/v1.15.14) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.13...v1.15.14) #### What's Changed - flate: Improve speed in big stateless blocks. by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#718 - zstd: Trigger BCE by switching on lengths by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#716 - zstd: Shave some instructions off the amd64 asm by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#720 - export NoGzipResponseWriter for custom ResponseWriter wrappers by [@​harshavardhana](https://togithub.com/harshavardhana) in [klauspost/compress#722 - s2: Add example for indexing and existing stream by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#723 - tests: Tweak fuzz tests by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#719 #### New Contributors - [@​harshavardhana](https://togithub.com/harshavardhana) made their first contribution in [klauspost/compress#722 **Full Changelog**: klauspost/compress@v1.15.13...v1.15.14 ### [`v1.15.13`](https://togithub.com/klauspost/compress/releases/tag/v1.15.13) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.12...v1.15.13) #### What's Changed - zstd: Add MaxEncodedSize to encoder by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#691 - zstd: Improve "best" end search by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#693 - zstd: Replace bytes.Equal with smaller comparisons by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#695 - zstd: Faster CRC checking/skipping by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#696 - zstd: Rewrite matchLen to make it inlineable by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#701 - zstd: Write table clearing in a way that the compiler recognizes by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#702 - zstd: Use individual reset threshold by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#703 - huff0: Check for zeros earlier in Scratch.countSimple by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#704 - zstd: Improve best compression's match selection by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#705 - zstd: Select best match using selection trees by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#706 - zstd: sync xxhash with final accepted patch upstream by [@​lizthegrey](https://togithub.com/lizthegrey) in [klauspost/compress#707 - zstd: Import xxhash v2.2.0 by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#708 **Full Changelog**: klauspost/compress@v1.15.12...v1.15.13 ### [`v1.15.12`](https://togithub.com/klauspost/compress/releases/tag/v1.15.12) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.11...v1.15.12) #### What's Changed - zstd: Tweak decoder allocs. by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#680 - gzhttp: Always delete `HeaderNoCompression` by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#683 **Full Changelog**: klauspost/compress@v1.15.11...v1.15.12 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xMDkuMSIsInVwZGF0ZWRJblZlciI6IjM0LjE1NC4wIn0=-->
kodiakhq bot
pushed a commit
to cloudquery/plugin-pb-go
that referenced
this pull request
Aug 1, 2023
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | minor | `v1.15.15` -> `v1.16.7` | --- ### Release Notes <details> <summary>klauspost/compress (github.com/klauspost/compress)</summary> ### [`v1.16.7`](https://togithub.com/klauspost/compress/releases/tag/v1.16.7) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.6...v1.16.7) #### What's Changed - zstd: Fix default level first dictionary encode by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#829 - docs: Fix typo in security advisory URL by [@​vcabbage](https://togithub.com/vcabbage) in [klauspost/compress#830 - s2: add GetBufferCapacity() method by [@​GiedriusS](https://togithub.com/GiedriusS) in [klauspost/compress#832 #### New Contributors - [@​vcabbage](https://togithub.com/vcabbage) made their first contribution in [klauspost/compress#830 - [@​GiedriusS](https://togithub.com/GiedriusS) made their first contribution in [klauspost/compress#832 **Full Changelog**: klauspost/compress@v1.16.6...v1.16.7 ### [`v1.16.6`](https://togithub.com/klauspost/compress/releases/tag/v1.16.6) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.5...v1.16.6) #### What's Changed - zstd: correctly ignore WithEncoderPadding(1) by [@​ianlancetaylor](https://togithub.com/ianlancetaylor) in [klauspost/compress#806 - gzhttp: Handle informational headers by [@​rtribotte](https://togithub.com/rtribotte) in [klauspost/compress#815 - zstd: Add amd64 match length assembly by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#824 - s2: Improve Better compression slightly by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#663 - s2: Clean up matchlen assembly by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#825 #### New Contributors - [@​rtribotte](https://togithub.com/rtribotte) made their first contribution in [klauspost/compress#815 - [@​dveeden](https://togithub.com/dveeden) made their first contribution in [klauspost/compress#816 **Full Changelog**: klauspost/compress@v1.16.5...v1.16.6 ### [`v1.16.5`](https://togithub.com/klauspost/compress/releases/tag/v1.16.5) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.4...v1.16.5) #### What's Changed - zstd: readByte needs to use io.ReadFull by [@​jnoxon](https://togithub.com/jnoxon) in [klauspost/compress#802 - gzip: Fix WriterTo after initial read by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#804 #### New Contributors - [@​jnoxon](https://togithub.com/jnoxon) made their first contribution in [klauspost/compress#802 **Full Changelog**: klauspost/compress@v1.16.4...v1.16.5 ### [`v1.16.4`](https://togithub.com/klauspost/compress/releases/tag/v1.16.4) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.3...v1.16.4) #### What's Changed - s2: Fix huge block overflow by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#779 - s2: Allow CustomEncoder fallback by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#780 - zstd: Fix amd64 not always detecting corrupt data by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#785 - zstd: Improve zstd best efficiency by [@​klauspost](https://togithub.com/klauspost) and [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#784 - zstd: Make load(32|64)32 safer and smaller by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#788 - zstd: Fix quick reject on long backmatches by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#787 - zstd: Revert table size change by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#789 - zstd: Respect WithAllLitEntropyCompression by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#792 - zstd: Fix back-referenced offset by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#793 - zstd: Load source value at start of loop by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#794 - zstd: Shorten checksum code by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#795 - zstd: Fix fallback on incompressible block by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#798 - gzhttp: Suppport ResponseWriter Unwrap() in gzhttp handler by [@​jgimenez](https://togithub.com/jgimenez) in [klauspost/compress#799 #### New Contributors - [@​jgimenez](https://togithub.com/jgimenez) made their first contribution in [klauspost/compress#799 **Full Changelog**: klauspost/compress@v1.16.3...v1.16.4 ### [`v1.16.3`](https://togithub.com/klauspost/compress/releases/tag/v1.16.3) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.2...v1.16.3) **Full Changelog**: klauspost/compress@v1.16.2...v1.16.3 ### [`v1.16.2`](https://togithub.com/klauspost/compress/releases/tag/v1.16.2) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.1...v1.16.2) #### What's Changed - Fix Goreleaser permissions by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#777 **Full Changelog**: klauspost/compress@v1.16.1...v1.16.2 ### [`v1.16.1`](https://togithub.com/klauspost/compress/releases/tag/v1.16.1) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.0...v1.16.1) #### What's Changed - zstd: Speed up + improve best encoder by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#776 - s2: Add Intel LZ4s converter by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#766 - gzhttp: Add BREACH mitigation by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#762 - gzhttp: Remove a few unneeded allocs by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#768 - gzhttp: Fix crypto/rand.Read usage by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#770 - gzhttp: Use SHA256 as paranoid option by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#769 - gzhttp: Use strings for randomJitter to skip a copy by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#767 - zstd: Fix ineffective block size check by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#771 - zstd: Check FSE init values by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#772 - zstd: Report EOF from byteBuf.readBig by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#773 - huff0: Speed up compress1xDo by [@​greatroar](https://togithub.com/greatroar) in [klauspost/compress#774 - tests: Remove fuzz printing by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#775 - tests: Add CICD Fuzz testing by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#763 - ci: set minimal permissions to GitHub Workflows by [@​diogoteles08](https://togithub.com/diogoteles08) in [klauspost/compress#765 #### New Contributors - [@​diogoteles08](https://togithub.com/diogoteles08) made their first contribution in [klauspost/compress#765 **Full Changelog**: klauspost/compress@v1.16.0...v1.16.1 ### [`v1.16.0`](https://togithub.com/klauspost/compress/releases/tag/v1.16.0) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.15.15...v1.16.0) #### What's Changed - s2: Add Dictionary support by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#685 - s2: Add Compression Size Estimate by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#752 - s2: Add support for custom stream encoder by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#755 - s2: Add LZ4 block converter by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#748 - s2: Support io.ReaderAt in ReadSeeker by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#747 - s2c/s2sx: Use concurrent decoding by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#746 - tests: Upgrade to Go 1.20 by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#749 - Update all (command) dependencies by [@​klauspost](https://togithub.com/klauspost) in [klauspost/compress#758 **Full Changelog**: klauspost/compress@v1.15.15...v1.16.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4yNi4xIiwidXBkYXRlZEluVmVyIjoiMzYuMjYuMSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows converting compressed LZ4 blocks to S2 (or snappy) blocks without decompression.
LZ4 -> S2 seems to be same size on average.
LZ4 -> Snappy is usually worse.
Single threaded performance
Speed (excluding LZ4 encoding):
Assembly speed (amd64)
Reference compression speed:
Size comparisons (using Go lz4 encoder):