Web cache poisoning -- X-Forwarded-Scheme vs X-Forwarded-Proto

[directionless]

Hi! I believe there default rack behavior of forwarded_scheme creates a cache poisoning vulnerability in common configurations. This was initially disclosed almost a year ago in the blog post Cache Poisoning At Scale, and I am starting to see it used by pentesters. As I don't an issue or fix, I wanted to document it here.

In short, many CDNs use X-Forwarded-Proto to convey the original request protocol. But, rack prefers X-Forwarded-Scheme, this creates a simple opportunity for a cache poisoning attack.

This is a bit hard to show, but if you assume the rack URL has a CDN in front of it:

URL=https://rack.example.com/assets/image.png?cachebust=${RANDOM}

curl -i  $URL
# Observe that the `x-cache` response is MISS. 

curl -i  $URL
# Observe that the `x-cache` response is now HIT

##
## Attack
## 
URL=https://rack.example.com/assets/image.png?cachebust=${RANDOM}

curl -H 'x-forwarded-scheme: http' -i $URL
# Observe the 301 response, and `x-cache: MISS`

curl  -i $URL
# observe a cached 301

This is something I had to patch on our CDN, and it sounds like something many sites have had to patch as well. (The previously linked blog post lists several notable sites. Here is the hackerone discussion)

[ioquatix]

Do you have any suggestions about how we can fix it?

[jeremyevans]

The obvious approach to "fixing it" would be respecting x-forwarded-proto in preference to x-forwarded-scheme if both are present. Of course, doing that will break existing safe usage where the CDN/proxy uses x-forwarded-scheme (since then an attacker could specify x-forwarded-proto). There is no completely safe default. We would need to make this configurable, and probably choose whichever is more popular as the default (and I don't know which is more popular).

[matthewd]

I think the long term answer is #1423, but that only helps when relevant upstreams have also adopted it.

[directionless]

I was thinking about what an ideal solution is, unfortunately I suspect that any time you check multiple headers, the site
must set them all or it creates this kind of attack vector. That feels inherent in any kind of A or B or C situation.

This leads me to think that the best route is to decide what's more common, and drop support for the other. Possibly with a documented config/monkeypatch/etc to change the behavior.

[ioquatix]

I also want to mention that there might be some issues regarding how we handle - and _ in headers. It could be feasible attack to write X_Forwarded_Proto or something similar.

[dcheckoway]

Hello! Any chance of a patch release containing this fix? I see it's planned for 3.0.0, but my team and I would really prefer not to wait for 3.0.0 to land and stabilize. Any chance we could have a 2.2.3.2 with this fix included? That would be much appreciated if so!!

[jeremyevans]

I think there is very little chance of backporting this change, it's way too disruptive for a tiny version change. It's not a bug fix, it's a completely new feature that may require configuration.

[dcheckoway]

Gotcha, thanks for the quick reply anyway!