Issue with GZip compression?

[vilu]

Hi, we noticed an issue when upgrading from 0.23.4 to 0.23.6. Some clients started failing and blaming the gzip compression. One example would be

io.netty.handler.codec.compression.DecompressionException: CRC value mismatch.

[armanbilge]

This is probably due to 8d3c422 in #5366. This could be a problem with fs2's gzip.

[Ravenow]

i can confirm this on 0.22.7, 0.22.6 works fine

[armanbilge]

Thanks for reporting. Yep, that would be the similar #5368 that targeted 0.22.

[armanbilge]

If either of you has an opportunity, would you be able to make a replication? Obviously this is something that is slipping through both the fs2 and http4s test suites 😕

[ybasket]

I made a reproducer, unfortunately not as simple test case because it requires that another HTTP implementation is doing a request, so it spins up a http4s server and uses the AHC/Netty client to make a request to it:
Repo: https://github.com/ybasket/http4s-gzip-reproducer/
Main:

object Main extends IOApp with Http4sDsl[IO] {

  val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
    case req => IO.println(req.headers) >> Ok("Foo")
  }

  override def run(args: List[String]): IO[ExitCode] =
    EmberServerBuilder.default[IO]
      .withHost(ipv4"0.0.0.0")
      .withPort(port"8080")
      .withHttpApp(GZip(routes).orNotFound)
      .build
      .use { _ =>
        AsyncHttpClient.resource[IO](new DefaultAsyncHttpClientConfig.Builder().setCompressionEnforced(true).build()).use { client =>
          client.expect[String]("http://localhost:8080/").flatMap(IO.println)
        }.as(ExitCode.Success)
      }
}

Fails with:

[error] (run-main-0) fs2.CompositeFailure: Multiple exceptions were thrown (2), first io.netty.handler.codec.compression.DecompressionException: CRC value mismatch. Expected: 3413357502, Got: 187806654
[error] fs2.CompositeFailure: Multiple exceptions were thrown (2), first io.netty.handler.codec.compression.DecompressionException: CRC value mismatch. Expected: 3413357502, Got: 187806654
[error] 	at fs2.CompositeFailure$.apply(CompositeFailure.scala:58)
[error] 	at fs2.CompositeFailure$.apply(CompositeFailure.scala:45)
[error] 	at fs2.CompositeFailure$.$anonfun$fromResults$1(CompositeFailure.scala:88)
[error] 	at scala.util.Either.fold(Either.scala:190)
[error] 	at fs2.CompositeFailure$.fromResults(CompositeFailure.scala:88)
[error] 	at fs2.Stream.$anonfun$merge$16(Stream.scala:1909)
[error] 	at get @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:73)
[error] 	at complete @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:55)
[error] 	at flatMap @ fs2.Stream.$anonfun$merge$15(Stream.scala:1908)
[error] 	at get @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:73)
[error] 	at complete @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:55)
[error] 	at flatMap @ fs2.Stream.$anonfun$merge$14(Stream.scala:1907)
[error] 	at complete @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:55)
[error] 	at complete @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:55)
[error] 	at flatMap @ fs2.Stream.$anonfun$merge$5(Stream.scala:1906)
[error] 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
[error] 	at handleErrorWith @ fs2.Compiler$Target.handleErrorWith(Compiler.scala:160)
[error] 	at modify @ org.http4s.ember.server.internal.Shutdown$$anon$1.<init>(Shutdown.scala:82)
[error] 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
[error] 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
[error] 	at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
[error] Caused by: io.netty.handler.codec.compression.DecompressionException: CRC value mismatch. Expected: 3413357502, Got: 187806654
[error] 	at io.netty.handler.codec.compression.JdkZlibDecoder.verifyCrc(JdkZlibDecoder.java:475)
[error] 	at io.netty.handler.codec.compression.JdkZlibDecoder.readGZIPHeader(JdkZlibDecoder.java:391)
[error] 	at io.netty.handler.codec.compression.JdkZlibDecoder.decode(JdkZlibDecoder.java:213)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[error] 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[error] 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[error] 	at io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:343)
[error] 	at io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:264)
[error] 	at io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:171)
[error] 	at io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:160)
[error] 	at io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
[error] 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[error] 	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
[error] 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
[error] 	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[error] 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[error] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[error] 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[error] 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[error] 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
[error] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
[error] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
[error] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
[error] 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
[error] 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[error] 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[error] 	at java.base/java.lang.Thread.run(Thread.java:834)
[error] 	at get @ fs2.internal.Scope.isInterrupted(Scope.scala:423)
[error] 	at flatMap @ fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:902)
[error] 	at get @ fs2.internal.Scope.isInterrupted(Scope.scala:423)
[error] 	at flatMap @ fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:902)
[error] 	at get @ fs2.internal.Scope.isInterrupted(Scope.scala:423)
[error] 	at flatMap @ fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:902)
[error] 	at main$ @ reproducer.Main$.main(Main.scala:13)

[armanbilge]

Thank you very much!!! This is great.

[AL333Z]

I had a look at this.. and it's fun.
So, comparing the bytes produced by fs2 and the former http4s implementation:

fs2:    List(31, -117, 8, 2, 0, 0, 0, 0, 0, 7, -66, -77, 115, -53, -49, 7, 0, -63, 35, 62, -76, 3, 0, 0, 0)
former: List(31, -117, 8, 0, 0, 0, 0, 0, 0, 0,           115, -53, -49, 7, 0, -63, 35, 62, -76, 3, 0, 0, 0) 
                          ^                    ^                    
                          |                    |CRC16 (2B)
                          |header flags `FLG` (1B)
                                  val FTEXT: Byte = 1
                                  val FHCRC: Byte = 2
                                  ...

If you squeeze your eyes, you can see that the header flags byte FLG is set by the fs2 implementation, which is setting the FLG.FHCRC bit. Then, note that the CRC16 bytes (-66, -77) is correctly set.
Also, the former implementation in http4s was not providing that flag (FLG=0), and that's probably the reason why it wasn't failing before.

The spec says:

If FHCRC is set, a CRC16 for the gzip header is present,
immediately before the compressed data. The CRC16 consists
of the two least significant bytes of the CRC32 for all
bytes of the gzip header up to and not including the CRC16.
[The FHCRC bit was never set by versions of gzip up to
1.2.4, even though it was documented with a different
meaning in gzip 1.2.4.]

Now, it seems that the fs2 implementation is just fine (computing the CRC32 and getting the least significant bytes..), so I checked netty.
This is where the FHCRC is verified, and if you go to the definition, it seems it's reading 4 bytes and verifying against a CRC32, and not a CRC16.

I'm not too familiar with this stuff, so please take it with a grain of salt...
I think I'll raise an issue there.

[ybasket]

Nice research!

We've seen failures with native iOS clients as well, so if that's a bug on the client side of things, it's not only Netty. Which would probably lead to the question whether we need some configurability here to allow clients that don't support the full spec. Or is there maybe some RFC that discourages the CRC16 in HTTP contexts?

Edit: https://www.rfc-editor.org/rfc/rfc2616#section-3.5 could be read as such hint (specifying values of Content-Encoding):

gzip An encoding format produced by the file compression program
"gzip" (GNU zip) as described in RFC 1952 [25]. This format is a
Lempel-Ziv coding (LZ77) with a 32 bit CRC.

[AL333Z]

We've seen failures with native iOS clients as well, so if that's a bug on the client side of things, it's not only Netty.

Oh boy...
Given the FLG.FHCRC check is optional (we were passing 0 to that flag in the former implementation..), and given we still don't know for sure whether

• fs2 is doing the right thing
• the clients implemented that check wrong (this is just my guess by reading the spec and looking into what netty is doing, but I may be wrong)

maybe supporting a config that lets the client choose to enable that additional check or not may be a good tradeoff. Also, if it turns out the clients are actually wrong, it may take time to get them fixed.

Edit: https://www.rfc-editor.org/rfc/rfc2616#section-3.5 could be read as such hint (specifying values of Content-Encoding)

I think that sentence is more related to the trailer, which must always contain the CRC32 for gzip, as per https://www.ietf.org/rfc/rfc1952.txt - section 2.3.

[armanbilge]

Thanks for chasing this up, this is fantastic. And seeing that netty hasn't rejected your PR at face value I'd say you're definitely on to something 😉

IIUC this FHCRC flag is an optional (but not required?) feature that fs2 is opting-in to, but unfortunately many clients have a bugged implementation of it or perhaps don't even support at all (?).

So whether we like it or not, it seems to me we need to make configurable whether to enable/disable this feature/flag, and make the old behavior the default in http4s for compatibility. This will need to be a change in fs2, ideally on both its series 2.x and 3.x branches for http4s 0.22 and 0.23, respectively.

Relevant LOC:
https://github.com/typelevel/fs2/blob/5c8346ce3f68f3a3122f6d5a11dd37432c0afc34/core/jvm/src/main/scala/fs2/compression/CompressionPlatform.scala#L437
https://github.com/typelevel/fs2/blob/5c8346ce3f68f3a3122f6d5a11dd37432c0afc34/core/jvm/src/main/scala/fs2/compression/CompressionPlatform.scala#L466

[AL333Z]

Perfectly summarized.

So whether we like it or not, it seems to me we need to make configurable whether to enable/disable this feature/flag, and make the old behavior the default in http4s for compatibility. This will need to be a change in fs2, ideally on both its series 2.x and 3.x branches for http4s 0.22 and 0.23, respectively.

I can pick this, while I'm at it :)

[vilu]

An additional data point: the client-side issues we saw on iOS were limited to iOS 12. We could not reproduce the issue on iOS 13 onwards. I have tried to find additional information but since iOS is closed source I can't really know why however, it could play into the idea that it's a client-side issue.

[armanbilge]

Cool, just saw that netty/netty#11805 is merged. Great work!

[armanbilge]

@vilu @ybasket thanks to @AL333Z's fantastic work on multiple fronts, if you use http4s 0.23.6 with fs2 3.2-14-a2508ec it should fix your issues. Please give it a try and let us know, when you have a chance! Thank you both for reporting and creating a reproduction.

@Ravenow never fear, a backport to fs2 2.x for http4s 0.22 is on the way in typelevel/fs2#2701.

[armanbilge]

@Ravenow Now available for http4s 0.22 in fs2 2.5-2-0f99472.

[rossabaker]

Where does that leave us? Leave this open until we have fs2 releases with the fix, but no code change on our end?

[armanbilge]

Yes, the default behavior was changed in fs2 so we don't need any code changes here. Up to you when to close the issue :)

[rossabaker]

Eh, no further action on our part, and people have a (snapshot) solution toward the bottom of this. Closing.

Great job tracking this one down.

[armanbilge]

For those afraid of hashes, this fix is now available in fs2 3.2.3:
https://github.com/typelevel/fs2/releases/tag/v3.2.3

[armanbilge]

And now (finally!) in fs2 2.5.11 as well:
https://github.com/typelevel/fs2/releases/tag/v2.5.11