Requests on pyth in 3 doesn't work the same as it does in python 2

[jeffreydwalter]

I'm having a problem with making an OPTIONS request. If I run the following code in python 2.7 with requests 2.7.0 it works fine and I get a 204 back. If I run it with python 3.8.5 I get a 403:

import requests
from requests_toolbelt.utils import dump

def print_raw_http(response):
    data = dump.dump_all(response, request_prefix=b'', response_prefix=b'')
    print('\n' * 2 + data.decode('utf-8'))

headers = {
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NETGEAR/v1 (iOS Vuezone)',
}
session = requests.Session()
r = session.options('https://ocapi-app.arlo.com/api/auth', headers=headers)
print_raw_http(r)
r.raise_for_status()

$ python -V
Python 2.7.16

$ python3 -V
Python 3.8.5

$ pip freeze | grep requests
requests==2.7.0

$ pip3 freeze | grep requests
requests==2.7.0

python 2.7:

$ python blah.py 


OPTIONS /api/auth HTTP/1.1
Host: ocapi-app.arlo.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NETGEAR/v1 (iOS Vuezone)
Content-Length: 0


HTTP/1.1 204 No Content

python 3.8.5:

$ python3 blah.py 


OPTIONS /api/auth HTTP/1.1
Host: ocapi-app.arlo.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NETGEAR/v1 (iOS Vuezone)
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 0


HTTP/1.1 403 Forbidden

Works fine with cURL too:

curl -vvvv -X OPTIONS "https://ocapi-app.arlo.com/api/auth" --output --http1.1 --no-alpn --no-npn -H "Host: ocapi-app.arlo.com" -H "Connection: keep-alive" -H "Accept-Encoding: gzip, deflate" -H "Accept: */*" -H "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NETGEAR/v1 (iOS Vuezone)" -H "Content-length: 0"

> OPTIONS /api/auth HTTP/1.1
> Host: ocapi-app.arlo.com
> Connection: keep-alive
> Accept-Encoding: gzip, deflate
> Accept: */*
> User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NETGEAR/v1 (iOS Vuezone)
> Content-length: 0
> 
< HTTP/1.1 204 No Content

[timthelion]

When pasting code blocks it is much easier to read if you use three `` tics around the code block. See these docs for more information.

[jeffreydwalter]

Yes, I know how to use markdown. That was a copy/paste fail while doing this from my phone.

See here if you want a formatted version: #5801 (comment)

[sigmavirus24]

Requests 2.7.0 is ancient and unsupported

[jeffreydwalter]

@sigmavirus24 I'm a little confused about the versioning. According to pypi, requests is on 2.25.1? What is the lastest stable version of requests?

[sethmlarson]

@jeffreydwalter Your pip freeze in the original post indicates 2.7.0, not 2.25.1. Is this the case?

[jeffreydwalter]

@sethmlarson, I was using 2.7.0, but I just installed requests==2.25.1, and the problem persists.

[jeffreydwalter]

@sethmlarson @sigmavirus24 can you please reopen this issue. This issue is relevant and happens with 2.25.1 as well.

[jeffreydwalter]

Thank you. Now that we're back in business, any thoughts about what the problem might be? The sample script I provided illustrates the problem.

[sigmavirus24]

It seems like that requires auth. Can you reproduce this with some other website or just Arlo?

[jeffreydwalter]

It's an OPTION request. If you run my test script in python 2.7 (or curl) you'll see you get back a 2XX response and a 403 in python 3.x.

…

On Mon, May 24, 2021, 5:43 PM Ian Stapleton Cordasco < ***@***.***> wrote: It seems like that requires auth. Can you reproduce this with some other website or just Arlo? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5817 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEKOBZRXO2INN7JSKAW4STTPLI7TANCNFSM446AL4DQ> .

[nateprewitt]

So, if you look at the response you're receiving @jeffreydwalter, you can see you're getting a CAPTCHA challenge from Cloudflare (not Arlo) because you've been identified as a bot.

I did a couple quick checks and the issue appears to be related how Requests is interacting with Urllib3's cipher setting. Cloudflare is doing fingerprinting and anything using Requests on Python 3.8+ is getting auto blocked. I happened upon this finding by downgrading urllib3 to 1.24 which works. However, both 1.23 and 1.24.1 do not.

This is the commit that was added back to 1.24.1 which fixed accidental removal of setting a default cipher in 1.24.0. I'd need to sit down for a minute to work out what we're passing into urllib3 for Python 3.8. This issue isn't reproducible using urllib3 alone, so it's specific to Requests.

There's also a question of whether we want to be in the business of trying to circumvent safety measures on services that have chosen to set them up. I'll do some more investigation if I find time but this sits pretty low on the priority list.

[jeffreydwalter]

That's fair. I appreciate you illuminating the issue. I assume Arlo put that in place to block people from using python to automate management of their personal cameras. There are 3-4 python libraries out there that copied my library and all of them have been thwarted by this. I'd venture there are a couple thousand users with broken home automation. I'm happy to take a look at the code if you can point me in the right direction. If not and you don't think this will get fixed in a reasonable amount of time, I can rewrite my library to use httpx or something else. Thanks again!

…

On Mon, May 24, 2021, 6:49 PM Nate Prewitt ***@***.***> wrote: There's also a question of whether we want to be in the business of trying to circumvent safety measures on services that have chosen to set them up. I'll do some more investigation if I find time but this sits pretty low on the priority list. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5817 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEKOBYP73EY5J7A3ECLFA3TPLQXXANCNFSM446AL4DQ> .

[jeffreydwalter]

@nateprewitt I can confirm that downgrading my urllib3 to 1.24 makes the issue go away. I think this is worth digging into because I don't think it's just a case of CloudFlare blocking use of python or requests. There's probably a subtle protocol issue at play which is causing them to reject the requests.

[sethmlarson]

Yeah this isn't something that Requests can do much about, you're being detected as "requiring additional auth" but the goalposts for what individual services like Cloudflare will decide can change any day so closing this as not actionable by our team. Thanks for the report!

[jeffreydwalter]

@sethmlarson what are you saying? You guys made a change to urllib3 and/or requests that caused this issue. Downgrading to urllib3 to 1.24 fixes the issue. The thing you can do about it is, fix the regression...