POST method fails with urllib>=1.26.0 for https://android.clients.google.com/auth

[finkandreas]

Subject

This issue baffles me a little bit, because I cannot understand why it fails. Basically I am unable to login to my google account when I'm using urllib>=1.26.0. But let me first describe the environments that I am using and how they are setup:

Environment

I am using python-3.8.6
WORKING:

python -m venv venv_working
venv_working/bin/pip install gpapi
venv_working/bin/pip install urllib3==1.25.11

BROKEN:

python -m venv venv_broken
venv_broken/bin/pip install gpapi

Steps to Reproduce

import http.client
http.client.HTTPConnection.debuglevel = 1
import logging
import requests
from gpapi.googleplay import GooglePlayAPI

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

api = GooglePlayAPI()
api.login(email='my_email', password='my_password')

Expected Behavior

I would expect that both versions will allow to login to the google account

Actual Behavior

It works with urllib3-1.25.11 but fails with urllib3>=1.26.0 (The response is 403 'Error=BadAuthentication'). I am logging the data that is sent to the server and it looks correct. Actually one can reduce the test case once the EncryptedPasswd is known, and create a pure POST request with requests (and get gpapi out of the equation).

data = {
        'Email': 'my_email',
        'EncryptedPasswd': replace_with_encrypted_pw_data,
        'add_account': '1',
        'accountType': 'HOSTED_OR_GOOGLE',
        'google_play_services_version': '12688052',
        'has_permission': '1',
        'source': 'android',
        'device_country': 'en',
        'lang': 'en_US',
        'client_sig': '38918a453d07199354f8b19af05ec6562ced5788',
        'callerSig': '38918a453d07199354f8b19af05ec6562ced5788',
        'service': 'ac2dm',
        'callerPkg': 'com.google.android.gms'
        }
requests.post('https://android.clients.google.com/auth', data=data)

This request will work with urllib3-1.25.11 and give a 403 with urllib3>=1.26.0. It is absolutely unclear to me what is different, because the data that is sent to the server is according to the logging exactly the same, so I do not see what the difference between the two POST requests is.

[pquentin]

I can reproduce on macOS. I ran git bisect and the first bad commit is 688584d which sends the "http/1.1" extension during the TLS handshake.

I'm not the only one seeing a correlation between ALPN and BadAuthentication, see kiwiz/gkeepapi#69 (comment). For now it looks like https://android.clients.google.com/auth just likes to drop suspicious requests, and urllib3 1.26 became suspicious due to ALPN. But that's not the only issue, I'll continue to investigate.

[finkandreas]

I investigated it also a bit more. Additional to ALPN it seems to be also related to OP_NO_TICKET. In 1.26.0 it is set by default, while in 1.25.11 it is not. Modifying to not use ALPN and removing OP_NO_TICKET in the default opts, would make it work again. I guess there is not much that can be done in urllib3, seems more like a server setup that rejects that type of requests. Feel free to close it, since I do not think that urllib3 can/should be modified in any way.

[sigmavirus24]

I agree @finkandreas - I don't think there's a good way to "fix" this in urllib3.

[pquentin]

@crwilcox Would it be possible to report to the team in charge of https://android.clients.google.com/auth that urllib3/requests (ie. most of the Python ecosystem) can't authenticate anymore since urllib3 1.26.0? Each of this two changes is enough on its own to get BadAuthentication 100% of the time:

• Enabling OP_NO_TICKET (see Disable the use of session tickets on TLSv1.2 by default #1970)
• Setting ALPN protocols (see Add HTTP/1.1 ALPN support #1894)

[crwilcox]

@pquentin I took a moment and ran the repro. It seems to me that perhaps the payload is being manipulated? if I run the POST the EncryptedPasswd is the only change I can see in the logs between the two requests. To be sure to rule out the library, I captured the data generated by gpapi + urllib 1.25.11 and injected it for gpapi + urllib 1.26.0 and it worked great.

Also the following will repro the behavior, and doesn't involve gpapi at all.

import http.client
http.client.HTTPConnection.debuglevel = 1
import logging
import requests
import os
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

data = {'Email': 'crwilcox.test@gmail.com', 'EncryptedPasswd': 'NOT-REALLY-abcd1234===', 'add_account': '1', 'accountType': 'HOSTED_OR_GOOGLE', 'google_play_services_version': '11951438', 'has_permission': '1', 'source': 'android', 'device_country': 'en', 'lang': 'en_US', 'client_sig': '38918a453d07199354f8b19af05ec6562ced5788', 'callerSig': '38918a453d07199354f8b19af05ec6562ced5788'}
AUTH_URL = 'https://android.clients.google.com/auth'

resp = requests.post(AUTH_URL, data=data, verify=True, proxies=None)
print(resp)

Log For Working:

❯ python main\ copy.py
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): android.clients.google.com:443
send: b'POST /auth HTTP/1.1\r\nHost: android.clients.google.com\r\nUser-Agent: python-requests/2.25.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 477\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'Email=crwilcox.test%40gmail.com&EncryptedPasswd=NOT-REALLY&add_account=1&accountType=HOSTED_OR_GOOGLE&google_play_services_version=11951438&has_permission=1&source=android&device_country=en&lang=en_US&client_sig=38918a453d07199354f8b19af05ec6562ced5788&callerSig=38918a453d07199354f8b19af05ec6562ced5788'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/plain; charset=utf-8
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Wed, 09 Dec 2020 20:54:26 GMT
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked
DEBUG:urllib3.connectionpool:https://android.clients.google.com:443 "POST /auth HTTP/1.1" 200 None
<Response [200]>

Log for Not Working

❯ python main\ copy.py
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): android.clients.google.com:443
send: b'POST /auth HTTP/1.1\r\nHost: android.clients.google.com\r\nUser-Agent: python-requests/2.25.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 477\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'Email=crwilcox.test%40gmail.com&EncryptedPasswd=NOT-REALLY&add_account=1&accountType=HOSTED_OR_GOOGLE&google_play_services_version=11951438&has_permission=1&source=android&device_country=en&lang=en_US&client_sig=38918a453d07199354f8b19af05ec6562ced5788&callerSig=38918a453d07199354f8b19af05ec6562ced5788'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Content-Type: text/plain; charset=utf-8
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Wed, 09 Dec 2020 20:54:49 GMT
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked
DEBUG:urllib3.connectionpool:https://android.clients.google.com:443 "POST /auth HTTP/1.1" 403 None
<Response [403]>
 

[jkuebart]

@crwilcox I have adapted your example so it also works with urllib3-1.26.3 for me now:

#!/usr/bin/env python3

import http.client
import requests
import ssl
from urllib3.poolmanager import PoolManager

http.client.HTTPConnection.debuglevel = 1

# Certain ciphers cause Google to return 403 Bad Authentication.
CIPHERS = ":".join(
    [
        "ECDHE+AESGCM",
        "ECDHE+CHACHA20",
        "DHE+AESGCM",
        "DHE+CHACHA20",
        "ECDH+AES",
        "DH+AES",
        "RSA+AESGCM",
        "RSA+AES",
        "!aNULL",
        "!eNULL",
        "!MD5",
        "!DSS",
    ]
)

class SSLContext(ssl.SSLContext):
    def set_alpn_protocols(self, protocols):
        """
        ALPN headers cause Google to return 403 Bad Authentication.
        """
        pass

class AuthHTTPAdapter(requests.adapters.HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        """
        Secure settings from ssl.create_default_context(), but without
        ssl.OP_NO_TICKET which causes Google to return 403 Bad
        Authentication.
        """
        context = SSLContext()
        context.set_ciphers(CIPHERS)
        context.options |= ssl.OP_NO_SSLv2
        context.options |= ssl.OP_NO_SSLv3
        context.options |= ssl.OP_NO_COMPRESSION
        context.post_handshake_auth = True
        context.verify_mode = ssl.CERT_REQUIRED
        self.poolmanager = PoolManager(*args, ssl_context=context, **kwargs)

AUTH_URL = "https://android.clients.google.com/auth"

data = {
    "Email": "",
    "EncryptedPasswd": "",
    "add_account": 1,
}

session = requests.session()
session.mount(AUTH_URL, AuthHTTPAdapter())
token = session.post(AUTH_URL, data)
print(token)

[sigmavirus24]

@lucasknopp this is not the place to ask that question. We're not Node developers and we're not going to research this for you

[jkuebart]

I fixed this in my own case by capturing and reusing the Client Hello from Android. If any interested the JA3 is:

769,49195-49196-52393-49199-49200-52392-158-159-49161-49162-49171-49172-51-57-156-157-47-53,65281-0-23-35-13-16-11-10,23,0

So any package that can handle JA3 should be able to make a successful request.

https://github.com/salesforce/ja3

@89z this sounds interesting, thank you for the pointer.

Could you elaborate a little on how this could be used to set up a working HTTPAdapter? In my comment above I used certain ciphers and SSL options to produce working requests. Is it possible to do this automatically given a JA3 fingerprint? If so, how?