Configure HostHeaderCustomizer

[a1730]

Jetty version
10.0.6
Started ServerConnector@59349a26{[proxy], ([proxy], http/1.1, h2c)}{server:8080}
Java version
openjdk version "17" 2021-09-14
Question
We are trying to upgrade from Jetty 9 to Jetty 10. Jetty is behind HAProxy in our deployment and we leverage haproxy protocol to reduce the pain of getting real client-IP behind multiple intermediate proxies. Jetty 7,8, and 9 worked beautifully. However, Jetty 10 insists on returning the user to the intermediate proxy. That was when we found out about the HostHeaderCustomizer.

So, we created simple module that adds the customizer to our server. The module worked in our test environment but it has no effect on the real McCoy leading us to believe that we are missing something. What should we be doing? We do not use the http-forwarded module in our environment.

http-customizer.mod

[description]
Enables HostHeaderCustomizer.

[tags]
connector

[depend]
http

[xml]
etc/jetty-http-customizer.xml

jetty-http-customizer.xml

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://www.eclipse.org/jetty/configure_10_0.dtd">
<Configure id="httpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
<Call name="addCustomizer">
<Arg>
<New class="org.eclipse.jetty.server.HostHeaderCustomizer"/>
</Arg>
</Call>
</Configure>

[joakime]

Jetty is behind HAProxy in our deployment and we leverage haproxy protocol to reduce the pain of getting real client-IP behind multiple intermediate proxies. Jetty 7,8, and 9 worked beautifully.

What does this mean? Are you using something like Proxy Protocol V2 or similar?

However, Jetty 10 insists on returning the user to the intermediate proxy.

Can you explain this in a bit more detail?

That hints at a Request Authority issue, either as a bug in ForwardRequestCustomizer, or as a configuration error in your HttpConfiguration.
I'd like to know the IP addresses of everything involved, your HttpConfiguration details, and the Forwarded header details you are getting from your haproxy, and finally what is happening in regards to the HTTP exchange (what you expect vs what actually happens. even showing Jetty 9 and Jetty 10 differences in the HTTP exchange would be helpful)

That was when we found out about the HostHeaderCustomizer.

This HostHeaderCustomizer was created to bridge the behavior differences with regards to request authority on HTTP/2 and older HTTP/1.0 clients to Web Applications that assume all requests have a Host header (they only do on HTTP/1.1). So unless you are using HTTP/2 or HTTP/1.0 this customizer is of little use for you.

The HostHeaderCustomizer will use the details from the Request object, obtained from the Connector, it's HttpConfiguration, SNI details in TLS, and even virtual host details.

As you can see, the use of HostHeaderCustomizer is most correct when your configuration is accurate and the Forwarded header from your haproxy is behaving properly. (Hint: do not use X-Forwarded-* headers, they are not a standard and have multiple different meanings that can often cause bad behavior. You should be using the standard RFC 7239 Forwarded header for the greatest odds of success)

[joakime]

I still don't understand how you managed to make this work in Jetty 7, 8, or 9 without the ForwardedRequestCustomizer as that is the only thing modifying the client details in those versions of Jetty.

I'm talking about the APIs ..

• HttpServletRequest.getRemoteAddr()
• HttpServletRequest.getRemoteHost()
• HttpServletRequest.getRemotePort()

Those values come from 1 of two places.
The raw connection details (in other words the raw TCP/IP Socket src address), or something using the the Request.setRemoteAddress(InetSocketAddress) API, which is only called from one (non-test) place.

The ForwardedRequestCustomizer

Here it is in Jetty 9.4.44
https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-server/src/main/java/org/eclipse/jetty/server/ForwardedRequestCustomizer.java#L535-L540

https://github.com/eclipse/jetty.project/blob/8da83308eeca865e495e53ef315a249d63ba9332/jetty-server/src/main/java/org/eclipse/jetty/server/ForwardedRequestCustomizer.java#L535-L540

[a1730]

I still don't understand how you managed to make this work in Jetty 7, 8, or 9 without the ForwardedRequestCustomizer as that is the only thing modifying the client details in those versions of Jetty.

I'm talking about the APIs ..

* `HttpServletRequest.getRemoteAddr()`
* `HttpServletRequest.getRemoteHost()`
* `HttpServletRequest.getRemotePort()`

Those values come from 1 of two places. The raw connection details (in other words the raw TCP/IP Socket src address), or something using the the Request.setRemoteAddress(InetSocketAddress) API, which is only called from one (non-test) place.

The ForwardedRequestCustomizer

Here it is in Jetty 9.4.44 https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-server/src/main/java/org/eclipse/jetty/server/ForwardedRequestCustomizer.java#L535-L540

https://github.com/eclipse/jetty.project/blob/8da83308eeca865e495e53ef315a249d63ba9332/jetty-server/src/main/java/org/eclipse/jetty/server/ForwardedRequestCustomizer.java#L535-L540

Hi @joakime,
I do not know why/how it works but please see our configuration from start.jar --list-config jetty9.txt. I generated that file from our production environment running Jetty 9. I hope it proves that we do not use the ForwardedRequestCustomizer (at least not deliberately.)

The layout in this setup is as follows:
human -> Cloudflare(reverse proxy) -> load balancer (HAProxy reverse proxy) -> prodsvcs(10.x.x.x).
Jetty 9 works. Jetty 10 keep sending the response to 10.x.x.x.

I will be able to get an equivalent Jetty 10 configuration tonight after hours when I can switch JETTY_HOME for a few seconds.

[joakime]

Thank you for that --list-config output, it helps focus the issue greatly.

Hmm, I think this might be the technique you are using.

${jetty.home}/etc/jetty-proxy-protocol.xml

That would be the file in your ${jetty.home} (thank you for using the ${jetty.base} and ${jetty.home} split properly btw).

It looks like this on Jetty 9.4.44

https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-server/src/main/config/etc/jetty-proxy-protocol.xml

https://github.com/eclipse/jetty.project/blob/8da83308eeca865e495e53ef315a249d63ba9332/jetty-server/src/main/config/etc/jetty-proxy-protocol.xml#L1-L9

That adds support to the http connector to support PROXY V1 and PROXY V2 based protocols.
Internally to Jetty, that basically results in a connection that uses the information given to it via the PROXY protocol for the TCP/IP src address.

This exists, in pretty much the same way, on Jetty 10.0.7

https://github.com/eclipse/jetty.project/blob/jetty-10.0.7/jetty-server/src/main/config/etc/jetty-proxy-protocol.xml

So lets focus on this PROXY protocol for a bit, as this is the likely path you were using before.

On your both your functional Jetty 9 server, and again on your problematic Jetty 10 server, can you capture DEBUG level logging for the named logger org.eclipse.jetty.server.ProxyConnectionFactory and get it to us?

That might reveal what's going on with your bad client / remote / src address.

[joakime]

The layout in this setup is as follows: human -> Cloudflare(reverse proxy) -> load balancer (HAProxy reverse proxy) -> prodsvcs(10.x.x.x). Jetty 9 works. Jetty 10 keep sending the response to 10.x.x.x.

What do you mean exactly when you say "Jetty 10 keeps sending the response to 10.x.x.x"?
Jetty is on the receiving end here, it doesn't initiate anything out to another site in your configuration. (you have no reverse proxy or forward proxy on Jetty defined at the level of --list-config)

Are you perhaps referring to your application responding with a redirect (like a 302 redirect) but it has a response header Location that contains the wrong IP address?

[a1730]

The layout in this setup is as follows: human -> Cloudflare(reverse proxy) -> load balancer (HAProxy reverse proxy) -> prodsvcs(10.x.x.x). Jetty 9 works. Jetty 10 keep sending the response to 10.x.x.x.

What do you mean exactly when you say "Jetty 10 keeps sending the response to 10.x.x.x"? Jetty is on the receiving end here, it doesn't initiate anything out to another site in your configuration. (you have no reverse proxy or forward proxy on Jetty defined at the level of --list-config)

Are you perhaps referring to your application responding with a redirect (like a 302 redirect) but it has a response header Location that contains the wrong IP address?

I meant that the browser was getting a wrong Location header of 10.x.x.x and the user becomes stuck. The only change between the two experiments is the value of JETTY_HOME (9.4.44 vs 10.0.7)

[a1730]

Please see attached output from --list_config from jetty-10 jetty10.txt, and the debug of the proxy-protocol handler. debug.txt
Thank you.

[joakime]

@a1730 the SameFileAliasChecker (Jetty 9) was replaced with AllowResourceAliasChecker (Jetty 10+) as part of Issue #6497 and PR #6668

[a1730]

Issue #7958 announcement (Jetty 9.x losing community support) forced us to deall with this issue.

Our configuration is end-user-> Cloudflare-> Haproxy-> Jetty server. Each with its own IP address.
These two lines in our Haproxy configuration worked for us.

http-request add-header Forwarded "for="%[src]"; proto=https"
http-request add-header X-Forwarded-Server "%[req.hdr(host)]"

%[src] is client real ip address, and %[req.hdr(host)] is the Host copied from HTTP request header. We noticed that Jetty tries to respond to the Haproxy address if we removed X-Forwarded-Server from our Haproxy configuration.

PS: Jetty is configured with haproxyprotocol v2, httpforward, and HTTP2 (H2C.)
Started ServerConnector@db5c3ab{[proxy], ([proxy], http/1.1, h2c)}{servername:8080}

Thank you for Jetty server.

[sbordet]

You should not be needing 2 lines, as Forwarded: for=%[src];proto=https;host=%[req.hdr(host)] should be enough.
Have you tried the host parameter and it does not work?

[a1730]

You should not be needing 2 lines, as Forwarded: for=%[src];proto=https;host=%[req.hdr(host)] should be enough. Have you tried the host parameter and it does not work?

That was what we thought. In fact that was why I decided to leave a comment ...
Reading the RFC, and following this haproxy thread for RFC7239 with Haproxy, we tried all imaginable combinations including adding host and server tokens. Nothing worked without adding http-request add-header X-Forwarded-Server "%[req.hdr(host)]".
We then started removing tokens until we were satisfied that "for", and "proto" were sufficient.

Our config?

Jetty Active XMLs:

/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-bytebufferpool.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-threadpool.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-webapp.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-deploy.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-gzip.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-http.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-http2c.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-http-forwarded.xml
/home/cglcons/vendor/jetty-home-10.0.9/etc/jetty-proxy-protocol.xml
${jetty.base}/etc/jetty-requestlog.xml

And

# Module: http-forwarded
# Enables processing of the "Forwarded" HTTP header (and its predecessors "X-Forwarded-" HTTP headers).
# The "Forwarded" HTTP header is added by intermediaries to provide information about the clients.
# ---------------------------------------
--module=http-forwarded
### ForwardedRequestCustomizer Configuration`
## Whether to process only the RFC7239 "Forwarded" header.
## "X-Forwarded-" headers are not processed.
jetty.httpConfig.forwardedOnly=true
#