Don’t know why server doesn’t accept aiohttp requests

[Pyprohly]

The following aiohttp code results in a 429 HTTP error.

import asyncio
import aiohttp

url = "http://www.reddit.com/.json?limit=1"
headers = {'User-Agent': 'asdf', 'Connection': 'close'}

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers=headers) as resp:
            print(await resp.text())
            print(resp.status)

asyncio.run(main())

But using requests and httpx works fine.

url = "http://www.reddit.com/.json?limit=1"
headers = {'User-Agent': 'asdf', 'Connection': 'close'}

import requests, httpx
for pkg in (requests, httpx):
    resp = pkg.get(url, headers=headers)
    print(resp.text)
    print(resp.status_code)

Using command line tools like curl, wget, or ncat also work.

curl -L -H 'User-Agent: asdf' 'http://www.reddit.com/.json?limit=1'


wget -qO - --header='User-Agent: asdf' 'https://www.reddit.com/.json?limit=1'


ncat --ssl reddit.com 443 <<EOF
GET /.json?limit=1 HTTP/1.1
Host: www.reddit.com
User-Agent: asdf
Connection: close
Accept: */*
Accept-Encoding: identity

EOF

So what is different about aiohttp's request that makes reddit servers not like it and how do I get it to work?

[Dreamsorcerer]

Weird. The response says to contact them (if you scroll past all the badgers...), so maybe they can explain why they are rate limiting a single request. The response claims this is done by IP, but that clearly is not true.

[Dreamsorcerer]

Can you check exactly what is being sent on the HTTP connections for 3.6 and 3.9? See if there are any differences in the HTTP request.

e.g. You can run nc -l 8000 (e.g. installed in Ubuntu with the netcat-openbsd package) and then change the URL to http://localhost:8000 and run the example.

[tailoric]

with python 3.6:

GET / HTTP/1.1
Host: localhost:8000
User-Agent: Some custom test header
Accept: */*
Accept-Encoding: gzip, deflate

with python 3.9

GET / HTTP/1.1
Host: localhost:8000
User-Agent: Some custom test header
Accept: */*
Accept-Encoding: gzip, deflate

I see no difference in the requests being sent
I also tested it with the requests library which manages to get a 200 response from reddit on all versions.

GET / HTTP/1.1
Host: localhost:8000
User-Agent: Some custom test header
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

the only change here is that requests also sends a Connection: keep-alive header but adding that header to my code above for testing doesn't change the outcome for aiohttp

[Dreamsorcerer]

So, the difference may only be in the TLS version or similar (I'm not familiar with how to debug at this low level). Still something to ask the support email in that response, it's unlikely we'll be able to do anything here.

We've had something similar with Tesla's API. They use a firewall (akamai?) and it ended up sometimes blocking requests on really odd details that was impossible to track down (it was so bad, I think they even ended up blocking their own official app at one point).

[tailoric]

Yeah I agree, seems like it is something we need to ask reddit directly.
Seems something in their server configurations changed since this problem has been happening for at least 3 weeks (at least that has been when I noticed it)

Though I am not sure if I'll get an answer because I asked on the 19th of September why I was IP blocked (I thought that was the reason at the time) and haven't gotten a response yet.
I will forward another E-Mail and include a link to this discussion and see if I will get an answer.

[webknjaz]

@tailoric I think you may be able to see the TLS version and ciphers in the handshake if you record and open it via Wireshark. I don't remember, though, if the beginning of the handshake is encrypted or not, needs testing.