HTTP Response compression not working with request accept headers containing q= when using Jetty and WebFlux

[MatthijsEM]

TLDR; the server is not using compression when it should when the accept header with the mime-types contains mime-type; q=x.

Tested this with Spring Boot 2.1.7 using Jetty and WebFlux. When using non-reactive (so no WebFlux) the problem does not occur.

With compression enabled the application.properties contains mime-types e.g.

server.compression.mime-types=text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json

After the server has started using debugging the Jetty GzipHandler is added to the context and its list of included mime-types does contain the above list.

The exclude list (defaults) consists of
image/ief, image/vnd.wap.wbmp, image/jpeg, application/bzip2, image/x-portable-graymap, application/brotli, image/bmp, image/gif, image/x-icon, audio/midi, video/x-msvideo, image/x-xbitmap, application/x-rar-compressed, image/x-portable-bitmap, image/x-rgb, image/x-cmu-raster, application/gzip, audio/x-wav, audio/x-pn-realaudio, audio/basic, application/compress, audio/x-aiff, video/x.ms.asx, video/x.ms.asf, image/png, video/vnd.rn-realvideo, image/x-xwindowdump, image/jpeg2000, video/x-sgi-movie, audio/mpeg, image/xcf, video/mpeg, image/x-portable-pixmap, image/tiff, image/x-portable-anymap, image/x-xpixmap, application/zip, video/quicktime, application/x-xz, video/mp4

It looks like the GzipHandler is properly configured.

Browsers like Chrome and Edge will send an Accept header like
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, */*; q=0.8
or
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3

With trace logging enabled using the last Accept header the server log will print:

org.eclipse.jetty.server.Server          : REQUEST GET /test on HttpChannelOverHttp@7f13c683{r=1,c=false,c=false/false,a=DISPATCHED,uri=//127.0.0.1:8080/test,age=6}
...
o.e.j.server.handler.gzip.GzipHandler    : GzipHandler@25533bba{STARTED} handle Request(GET //127.0.0.1:8080/test)@28ed18ab in null```
...
o.s.w.r.r.m.a.ResponseBodyResultHandler  : Using 'application/json;q=0.8' given [text/html, application/xhtml+xml, image/webp, image/apng, application/signed-exchange;v=b3, application/xml;q=0.9, */*;q=0.8] and supported [application/json]
...
o.e.j.s.h.g.GzipHttpOutputInterceptor    : org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@17728af exclude by mimeType application/json;q=0.8

The last line already indicates that the response is not compressed while it should because the mime-type does not match. When debugging the GzipHandler isMimeTypeGzipable returns false because its trying to match application/json;q=0.8 against application/json.

The response is not compressed but it should be.

The q=x indicates a quality factor, the preference when sending multiple mime-types to the server so the server can choose based on the client preference. The suffix itself should not be passed to the GzipHandler which is not capable of handling it.

[philwebb]

@MatthijsEM Are you able to share the sample application that you used to test this?

[bclozel]

I've tried to reproduce this issue and couldn't.

1. I've created a Spring Boot 2.2.2.RELEASE application on start.spring.io with the webflux starter
2. added the jetty starter to the POM
3. added a src/main/resources/static/test.txt file with random content in it
4. added the following to application.properties:

server.compression.enabled=true
server.compression.min-response-size=128B
logging.level.org.eclipse.jetty.server.handler.gzip=TRACE

The following curl command shows that the response is indeed compressed:

curl -H "Accept-Encoding: gzip" -H "Accept: text/plain; q=0.9" -I localhost:8080/test.txt
HTTP/1.1 200 OK
Content-Type: text/plain
Accept-Ranges: bytes
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Content-Length: 20

And the logs are:

[ttp@2ef8a8c3-31] o.e.j.server.handler.gzip.GzipHandler    : GzipHandler@512d4583{STARTED,min=128,inflate=-1} handle Request(HEAD //localhost:8080/test.txt)@5b100e4c in null
[ttp@2ef8a8c3-31] o.e.j.s.h.g.GzipHttpOutputInterceptor    : org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@5fd59236 compressing java.util.zip.Deflater@83fd96c

I'm probably missing something here. Could you share a sample application reproducing this issue?
Thanks!

[MatthijsEM]

I was able to reproduce this using the exact setup you mentioned above (#18484 (comment)) but instead of a txt file I used a simple REST controller returning some JSON.

package com.example.demo;

import java.util.stream.Collectors;
import java.util.stream.IntStream;

import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import reactor.core.publisher.Mono;

@RestController
public class TestController {

	@GetMapping(path = { "/test" }, produces = MediaType.APPLICATION_JSON_VALUE)
	public Mono<String> test() throws Exception {

		return Mono.just("\"" + IntStream.range(0, 1024).mapToObj(String::valueOf).collect(Collectors.joining()) + "\"");
	}
}

When calling this using

curl -H "Accept-Encoding: gzip" -H "Accept: */*" -I localhost:8080/test

the server returns

HTTP/1.1 200 OK
Date: Fri, 20 Dec 2019 09:58:54 GMT
Content-Type: application/json;charset=utf-8
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Content-Length: 20

which has compression and is correct.

When calling this with

curl -H "Accept-Encoding: gzip" -H "Accept: application/json; q=0.9" -I localhost:8080/test

or more standard something like

curl -H "Accept-Encoding: gzip" -H "Accept: text/plain; q=0.9, */*; q=0.8" -I localhost:8080/test

the server returns

HTTP/1.1 200 OK
Date: Fri, 20 Dec 2019 10:07:34 GMT
Content-Type: application/json;q=0.9;charset=UTF-8
Content-Length: 2988

with no compression which is wrong.

The server log will show

2019-12-20 12:34:56.655 DEBUG 16456 --- [ttp@304a9d7b-21] o.e.j.s.h.g.GzipHttpOutputInterceptor : org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@361ef3be exclude by mimeType application/json;q=0.9

[bclozel]

Thanks for the sample, I can see that now.

In Jetty's GzipHttpOutputInterceptor, the response Content-Type is retrieved from the headers and compared against the whitelisted media types for gzipping. In this case, it's getting the full Content-Type and only removing the charset parameter:

requestEtags = MimeTypes.getContentTypeWithoutCharset(requestEtags);

This means that Jetty will not remove any other media type parameter from the Content-Type, so it might try and compare application/json;param=value against the whitelisted application/json.

There's nothing we can do about that at the Spring Boot level. Could you create an issue in the Jetty project? Note that it seems that they don't want to expand features there too much, but I think that removing media type parameters before comparing with the whitelisted types is expected.

In the meantime, WebFlux should not echo back the quality parameter in the response, I've created spring-projects/spring-framework#24239 to fix that.

Thanks!