reproducible HTTP 502 error in OAuth /oauth/authorize

[snarfed]

Hi all! I keep hitting a 502 in /oauth/authorize in 3.0.1. Is this a known issue? Here are the steps to reproduce:

1. Create an app, eg:

   POST https://mastodon.technology/api/v1/apps
   client_name=Bridgy&redirect_uris=http://localhost:8080/mastodon/delete/finish%0Ahttp://localhost:8080/delete/finish%0Ahttp://localhost:8080/publish/mastodon/finish%0Ahttp://localhost:8080/mastodon/callback&website=http://localhost:8080&scopes=read+read:accounts+read:blocks+read:favourites+read:filters+read:follows+read:lists+read:mutes+read:notifications+read:reports+read:search+read:statuses+write+write:accounts+write:blocks+write:favourites+write:filters+write:follows+write:lists+write:media+write:mutes+write:notifications+write:reports+write:statuses+follow+push
   

   (I've URL-unescaped everything except the newlines, here and below, for readability.)
2. Authorize with an initial set of read only scopes, eg:

   https://mastodon.technology/oauth/authorize?response_type=code&client_id=...&client_secret=...&scope=read:accounts read:accounts read:blocks read:search&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state={"app_key":"...","state":"%257B%2522feature%2522%253A%2522listen%2522%252C%2522operation%2522%253A%2522add%2522%257D"}
   

   ...and get an access token with /oauth/token.
3. Authorize with an expanded set of scopes, including writes, eg:

   https://mastodon.technology/oauth/authorize?response_type=code&client_id=...&client_secret=...&scope=read:accounts read:accounts read:blocks read:search write:statuses write:favourites write:media&redirect_uri=http://localhost:8080/mastodon/callback&state={"app_key":"...","state":"%7B%22feature%22%3A%22publish%22%2C%22operation%22%3A%22add%22%7D"}
   

   ...and get an access token with /oauth/token.
4. Authorize with the same expanded set of scopes and a longer state, eg:

   https://mastodon.technology/oauth/authorize?response_type=code&client_id=...&client_secret=...&scope=read:accounts read:accounts read:blocks read:search write:statuses write:favourites write:media&redirect_uri=http://localhost:8080/publish/mastodon/finish&state={"app_key":"...","state":"%7B%22include_link%22%3A%22omit%22%2C%22source_key%22%3A%22...%22%2C%22source_url%22%3A%22http%3A%2F%2Flocalhost%2Fpost3.html%22%2C%22target_url%22%3A%22https%3A%2F%2Fbrid.gy%2Fpublish%2Fmastodon%22%7D"}
   

This last request 502s with the standard We're sorry, but something went wrong on our end. error page.

I tried three 3.0.1 instances just now - 2020-01-21 05:36:12 UTC on mastodon.technology, 05:49:03 mas.to, 06:10:37 mstdn.social - and they all behaved the same way.

Any idea what's going on? My only guess is that the state in the last request is longer than some limit, but that seems unlikely. (It was originally 331 chars before I redacted parts of it.)

@ashfurrow, any chance you could look through the mastodon.technology logs around that time ^ and see if you can correlate the /oauth/authorize 502 with anything?

Thanks in advance!

[ashfurrow]

I took some time looking through the logs but couldn't find anything helpful, sorry!

[snarfed]

Hey @ashfurrow, I've been hitting this again recently, so I did some digging, and I can only reproduce it on mastodon.technology, not on other 3.4.1 instances like eg mastodon.art. So, I suspect it might be specific to your fork or config. Details below. Let me know if you want me to file an issue on https://github.com/ashfurrow/mastodon/issues!

It looks like there's an overall character limit on the URL or path or query string. For example, this URL correctly returns an error that the client id is invalid:

https://mastodon.technology/oauth/authorize?response_type=code&client_id=-------------------------------------------&client_secret=-------------------------------------------&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses%20write:statuses%20write:favourites%20write:media&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state=----------------------------------------------------------------------------------------------------------------------------------------------------------------------

...and if I fill in a valid client id and secret, it works. However, if I add a few characters, eg:

https://mastodon.technology/oauth/authorize?response_type=code&client_id=-------------------------------------------&client_secret=-------------------------------------------&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses%20write:statuses%20write:favourites%20write:media&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state=------------------------------------------------------------------------------------------------------------------------------------------------------------------------XXXX

...it 502s, even with a valid client id/secret. If I remove characters from the query string, regardless of where, the 502 goes away.

(Oddly, the exact limit seems a bit flaky. I'm hitting it around 551 chars, but it's not always exactly the same. Maybe it's overall HTTP request payload, including headers, which vary a bit due to timestamps or something similar?)

I can add many more characters to the end on eg mastodon.art, and it doesn't 502 the same way:

https://mastodon.art/oauth/authorize?response_type=code&client_id=-------------------------------------------&client_secret=-------------------------------------------&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses%20write:statuses%20write:favourites%20write:media&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state=-------------------------------------------------------------------------------------------------------------------------------------------------------------------------XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[snarfed]

Apologies, it may not be mastodon.technology specific after all. I see it on the other two instances I originally tested this on too, mas.to and mstdn.social, both 3.4.1. Here are the longest /oauth/authorize URLs that work on those two instances:

https://mstdn.social/oauth/authorize?response_type=code&client_id=-------------------------------------------&client_secret=-------------------------------------------&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses%20write:statuses%20write:favourites%20write:media&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state=--------------------------------------------------------------------------------------------------------------------------------

https://mas.to/oauth/authorize?response_type=code&client_id=-------------------------------------------&client_secret=-------------------------------------------&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses%20write:statuses%20write:favourites%20write:media&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state=-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Adding a character to the end of either one triggers the 502. (And again, it doesn't seem to be perfectly consistent, sometimes it's a few characters in either direction.)

Odd that I can't trigger it on mastodon.art though!

[snarfed]

My users are continuing to hit this on a growing number of interactions with Mastodon OAuth, now sometimes even on initial login. I may look at debugging and fixing it myself soon.

[snarfed]

Haven't yet been able to reproduce this against a local Mastodon server built from HEAD. It handled this 785-character URL fine:

http://localhost:3000/oauth/authorize?response_type=code&client_id=2DgqdtD0K9EU2_o3RFEOdLwr6UwoL_92ZmrQCl4sB1g&client_secret=...&scope=read:accounts%20read:blocks%20read:bookmarks%20read:favourites%20read:filters%20read:follows%20read:lists%20read:mutes%20read:notifications%20read:search%20read:statuses%20write%20write:accounts%20write:blocks%20write:bookmarks%20write:favourites%20write:filters%20write:follows%20write:lists%20write:media%20write:mutes%20write:notifications%20write:reports%20write:statuses%20follow%20push&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fmastodon%2Fcallback&state={"app_key":"agNhcHByEQsSC01hc3RvZG9uQXBwGAEM","state":"%257B%2522feature%2522%253A%2522listen%2522%252C%2522operation%2522%253A%2522add%2522%257D"}

[snarfed]

Production Mastodon instances still 502 though, eg mastodon.technology:

https://mastodon.technology/oauth/authorize?response_type=code&client_id=...&client_secret=...&scope=read:accounts%20read:blocks%20read:notifications%20read:search%20read:statuses&redirect_uri=https%3A%2F%2Fbrid.gy%2Fmastodon%2Fcallback&state=%7B%22app_key%22:%22agdicmlkLWd5chgLEgtNYXN0b2RvbkFwcBiAgKjUhuuPCgw%22,%22state%22:%22%257B%2522feature%2522%253A%2522listen%2522%252C%2522operation%2522%253A%2522add%2522%257D%22%7D

(Not that exact URL, of course, since I've redacted the client id and secret. Happy to provide the exact URL somewhere non-public if anyone wants to dig in.)

[metbril]

I have a similar issue since this morning with mastodon.social running 3.4.6.

[Gargron]

I suspect that it doesn't happen on localhost because the error is returned by nginx. Mastodon does not return HTTP 502 errors on its own. HTTP 502 means "Bad gateway" and means nginx could not get or process the response from the gateway (Mastodon).

I believe I found the corresponding error in /var/log/nginx/error.log:

upstream sent too big header while reading response header from upstream

[Gargron]

The "solution" is to increase the size of the proxy buffers in nginx, but this isn't really a distributable solution, and I'm concerned about the implications on other types of requests. It seems like it could be a mistake on behalf of the doorkeeper gem in sending back a header that's way too long.

[snarfed]

Thank you for investigating!

[mlncn]

Any mitigations identified (even to go from very flaky to somewhat flaky) or possible for the workaround nginx proxy buffer increase to be done for mastodon.social ? Currently blocking all signup attempts from Bridgy

EDIT: It repeatedly failed when trying from Firefox (container) where i was logged in to Mastodon.social, but it worked in Chromium where i was not logged in. Confirmed that simply logging out before trying to authorize was a consistent workaround for me, regardless of browser/container.

[Nezteb]

Can confirm @mlncn's workaround:

1. Go to https://mastodon.social/
2. Sign out of Mastodon
3. Go to https://brid.gy/
4. Click on the Mastodon button
5. Sign into Mastodon
6. Enjoy your sweet sweet Bridgy/Mastodon connection

cc snarfed/bridgy#911

[snarfed]

@Nezteb @mlncn confirmed, that workaround works for me too! I'll follow up in doorkeeper-gem/doorkeeper#1554

[thiagomgd]

workaround was working for me, but stopped over the weekend. Tested on 2 computers (both macOs, chrome), on regular window and also incognito

[timnolte]

The "solution" is to increase the size of the proxy buffers in nginx, but this isn't really a distributable solution, and I'm concerned about the implications on other types of requests. It seems like it could be a mistake on behalf of the doorkeeper gem in sending back a header that's way too long.

I can confirm that using the proxy buffer settings mentioned here for my Mastodon instance Nginx configuration did indeed fix the 502 errors.

[snarfed]

My (and other) users keep hitting this and getting confused and blocked by it. It's preventing a sizeable fraction of them from using my service at all.

@Gargron @ClearlyClaire and other maintainers, I understand that you're reluctant to increase the nginx proxy buffer sizes, but as people have confirmed here and in doorkeeper-gem/doorkeeper#1554 (comment), it clearly does fix this problem. The lengths of the redirect URLs and headers from doorkeeper seem correct, not a mistake. Could we increase the proxy buffers a moderate amount, so that it's hopefully not too invasive but still solves the problem?

[snarfed]

I ended up working around this by storing the OAuth redirect state in memory on my end, which let me reduce the redirect URL length. I'll leave this open to track on Mastodon's end.

[33b5e5]

I can confirm that using the proxy buffer settings mentioned here for my Mastodon instance Nginx configuration did indeed fix the 502 errors.

Thank you. I added these three lines:

proxy_buffers 4 16k;
proxy_buffer_size 16k;
proxy_busy_buffers_size 32k;

I think they could technically go in a http, server, or location block, but I chose to add them to the location @proxy block that proxies requests to http://backend. My thinking is this will cover the endpoints exhibiting the problem, while limiting the impact of the change otherwise.