Feature: Add option to force compressed data to be byte aligned #49

ryancdotorg · 2020-11-22T14:05:26Z

This adds byte_align and bare_stream parameters to enable production of Brotli compressed blocks that can be used to construct a complete stream with trivial byte-wise concatonation.

To my knowledge, this functionality does not currently exist in any other Brotli implementation.

These options are exposed via the command line tool as --bytealign and --bare (bare mode enables byte alignment).

Byte align mode inserts an empty metadata block before the final empty block if the compressed data block is not already byte aligned.

Bare mode additionally omits the final empty block, and, in catable mode, the stream header.

TODO

don't emit empty penultimate metadata block if the compressed block ended on a byte boundry
add support for emitting "bare" compressed data without a stream header or empty last block
don't bother with byte aligning the start in appendable mode
plumb for use by libraries
update documentation
add tests

Notes to maintainers

This is the first time I've ever touched Rust code, so please forgive me if I've done something silly.

ryancdotorg · 2020-12-15T12:46:50Z

@philippeitis @danielrh can i get some feedback on this please?

philippeitis · 2020-12-16T22:10:16Z

I'm not actually a maintainer, but my feedback is largely nit-picking code style. However, given that the codebase itself was machine-generated, I don't think that's a major issue. Otherwise, I think providing a link in the PR to relevant documentation of this feature in other brotli implementations (if it exists, otherwise saying that it's original would also be fine) would be helpful. Again, not a maintainer, so take my comments with a grain of salt.

philippeitis · 2020-12-16T22:14:21Z

src/enc/encode.rs

@@ -340,6 +341,17 @@ value: u32) -> i32 {
    params.favor_cpu_efficiency = value != 0;
    return 1i32;
  }
+  if p as (i32) == BrotliEncoderParameter::BROTLI_PARAM_BYTE_ALIGN as (i32) {


I think if p == BrotliEncoderParameter::BROTLI_PARAM_BYTE_ALIGN should compile on rust 1.12? (I know that the rest of the code in this function uses the other convention, but from what I understand, this is all machine generated, and not how you'd naturally write the code).

philippeitis · 2020-12-16T22:17:54Z

src/enc/encode.rs

@@ -721,7 +740,9 @@ fn EnsureInitialized<Alloc: BrotliAlloc>
    if (*s).params.quality == 0i32 || (*s).params.quality == 1i32 {
      lgwin = brotli_max_int(lgwin, 18i32);
    }
-    EncodeWindowBits(lgwin, s.params.large_window, &mut (*s).last_bytes_, &mut (*s).last_bytes_bits_);
+    if !(*s).params.bare_stream {
+      EncodeWindowBits(lgwin, s.params.large_window, &mut (*s).last_bytes_, &mut (*s).last_bytes_bits_);


We don't need to do &mut (*s).field - &mut s.field works identically and looks nicer (again, machine generated code which isn't very natural).

philippeitis · 2020-12-16T22:19:58Z

src/enc/encode.rs

@@ -2014,7 +2035,13 @@ fn WriteMetaBlockInternal<Alloc: BrotliAlloc,
                                     false,
                                     cb);
    if actual_is_last != is_last {
-      BrotliWriteEmptyLastMetaBlock(storage_ix, storage)
+      // insert empty block for byte alignment if required
+      if params.byte_align && ((*storage_ix & 7u32 as (usize)) != 0) {


Better as (storage_ix & 7) != 0 (machine generated code)

philippeitis · 2020-12-16T22:20:08Z

src/enc/encode.rs

@@ -2154,7 +2181,13 @@ fn WriteMetaBlockInternal<Alloc: BrotliAlloc,
                                       cb);
  }
  if actual_is_last != is_last {
-    BrotliWriteEmptyLastMetaBlock(storage_ix, storage)
+    // insert empty block for byte alignment if required
+    if params.byte_align && ((*storage_ix & 7u32 as (usize)) != 0) {


Better as (storage_ix & 7) != 0 (machine generated code)

ryancdotorg · 2020-12-16T22:38:40Z

Thanks for following up. I was basing my additions on the existing codebase, I didn't realize it was machine generated. I think it'd be mildly confusing to mix use of e.g. (*s).field and s.field, but not my repo, and I'm find doing it however will get merged.

To the best of my knowledge, this feature doesn't exist in any other Brotli implementation, and I can edit the PR description to reflect that a bit later. I can expand a bit on the motivation as well.

danielrh · 2021-07-31T20:39:50Z

Hi! Thanks for doing this--sorry I've been away for a bit but I'm back now...I'd love to hear about the motivation of this patch and what problem it's trying to solve...is it taking the catable command one step further to allow vanilla cat instead of the special broccoli cat tools to combine files?

ryancdotorg · 2021-08-02T03:43:17Z

@danielrh Yes, the idea is to produce substreams which can be assembled with the generic cat command or other tools that do naive concatenation/copying of bytes. Simply copying bytes ought to be faster than having to do even minor bit shuffling, though I feel like simplicity is a bigger win here.

This could be used in a number of scenarios, but the two main example I had in mind:

Stitching together multiple blocks of precompressed HTML, JavaScript and/or CSS into a stream
Limited templating with precompressed data, e.g. a precompressed JavaScript IIFE with an object literal embedded at the very end as a data parameter

ryancdotorg · 2021-10-18T04:57:15Z

What do we want to do about the the issue of changes matching existing code conventions vs changes being reasonably idiomatic rust code? My sensibilities swing towards keeping code style consistent, but it's your call.

CLAassistant · 2022-04-16T21:53:04Z

All committers have signed the CLA.

johnterickson · 2023-11-09T20:28:10Z

Ok this is really cool! We have a scenario that is similar to https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less where we our customers upload to a content-addressable-store.

Create a header block:
touch empty.bin
--appendable --bytealign --bare -c empty.bin start.br
Individually compress the actual data blocks like this:
--catable --bytealign --bare -c block001 block001.br
Then fake a ISLASTEMPTY metablock:
printf "\x03" > ~/end.br

Then you can both

Decompress individual blocks by a) prepending the start block and b) appending the end block: cat start.br block001.br end.br | brotli -d
A standard decompressor (e.g. curl --compressed) can recreate the whole file from the concatenation of all the compressed blocks: e.g. cat start.br block*.br end.br | brotli -d
If you need to, rearrange the compressed blocks order to rearrange the output order

johnterickson · 2023-11-12T15:43:09Z

@ryancdotorg Have you done anything more with this idea?

I brought it over to https://github.com/johnterickson/BrotliSharpLib/tree/bytealign

ryancdotorg · 2023-11-12T19:10:42Z

@johnterickson No, haven't been working on it. All this PR needs is a rebase, tests and updated docs...

ryancdotorg changed the title ~~bytealign mode: basic functionality~~ Feature: Add option to force compressed data to be byte aligned Nov 22, 2020

ryancdotorg force-pushed the bytealign branch 2 times, most recently from 2b03734 to 1596167 Compare November 22, 2020 18:14

philippeitis reviewed Dec 16, 2020

View reviewed changes

ryancdotorg and others added 6 commits November 8, 2021 18:47

bytealign mode: basic functionality

0e9d03c

don't emit empty penultimate metadata block unless needed

2877641

byte aligning the start of an appendable is pointless

34182ee

add option for bare stream

c44ecc7

no bytealign unless catable, appendable, or bare

0f746e5

try to plumb ffi apis

df6a88e

ryancdotorg force-pushed the bytealign branch from 799f10c to df6a88e Compare November 9, 2021 01:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Add option to force compressed data to be byte aligned #49

Feature: Add option to force compressed data to be byte aligned #49

ryancdotorg commented Nov 22, 2020 •

edited

ryancdotorg commented Dec 15, 2020

philippeitis commented Dec 16, 2020 •

edited

philippeitis Dec 16, 2020 •

edited

philippeitis Dec 16, 2020

philippeitis Dec 16, 2020

philippeitis Dec 16, 2020

ryancdotorg commented Dec 16, 2020

danielrh commented Jul 31, 2021

ryancdotorg commented Aug 2, 2021

ryancdotorg commented Oct 18, 2021

CLAassistant commented Apr 16, 2022 •

edited

johnterickson commented Nov 9, 2023 •

edited

johnterickson commented Nov 12, 2023

ryancdotorg commented Nov 12, 2023

Feature: Add option to force compressed data to be byte aligned #49

Are you sure you want to change the base?

Feature: Add option to force compressed data to be byte aligned #49

Conversation

ryancdotorg commented Nov 22, 2020 • edited

TODO

Notes to maintainers

ryancdotorg commented Dec 15, 2020

philippeitis commented Dec 16, 2020 • edited

philippeitis Dec 16, 2020 • edited

Choose a reason for hiding this comment

philippeitis Dec 16, 2020

Choose a reason for hiding this comment

philippeitis Dec 16, 2020

Choose a reason for hiding this comment

philippeitis Dec 16, 2020

Choose a reason for hiding this comment

ryancdotorg commented Dec 16, 2020

danielrh commented Jul 31, 2021

ryancdotorg commented Aug 2, 2021

ryancdotorg commented Oct 18, 2021

CLAassistant commented Apr 16, 2022 • edited

johnterickson commented Nov 9, 2023 • edited

johnterickson commented Nov 12, 2023

ryancdotorg commented Nov 12, 2023

ryancdotorg commented Nov 22, 2020 •

edited

philippeitis commented Dec 16, 2020 •

edited

philippeitis Dec 16, 2020 •

edited

CLAassistant commented Apr 16, 2022 •

edited

johnterickson commented Nov 9, 2023 •

edited