Investigate CI failures with Python 3.13.0a5

[illia-v]

Tests on all supported OSes started failing with recent Python 3.13.0a5.

Tests succeeded on Linux and macOS with Python 3.13.0a4 and on Windows with Python 3.13.0a3.

Let's investigate the errors and determine on who's side fixes are needed. If urllib3's code has to be updated to work with Python 3.13, we have to check the 1.26.x branch too.

https://github.com/urllib3/urllib3/actions/runs/8317963362/job/22759351029

[illia-v]

I was able to reproduce all 35 Ubuntu 22.04 (OpenSSL 3.0) 3.13 errors locally.

The 35 original errors

============================================================================== short test summary info ===============================================================================
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=35741): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=35741): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=45701): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=45701): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=40907): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=40907): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=41275): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=41275): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_basic_proxy - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_proxy - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_is_verified_http_proxy_to_http_target - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_is_verified_https_proxy_to_http_target - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_http_and_https_kwarg_ca_cert_data_proxy - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_proxy_with_proxy_ssl_context - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_proxy_pyopenssl_not_supported - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_proxy_forwarding_for_https - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_nagle_proxy - AttributeError: 'NoneType' object has no attribute 'getsockopt'
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_oldapi - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_redirect - assert 500 == 303
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_cross_host_redirect - Failed: DID NOT RAISE <class 'urllib3.exceptions.MaxRetryError'>
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_cross_protocol_redirect - Failed: DID NOT RAISE <class 'urllib3.exceptions.MaxRetryError'>
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_headers - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_headers - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_https_headers_forwarding_for_https - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_headerdict - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPProxyManager::test_scheme_host_case_insensitive - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestIPv6HTTPProxyManager::test_basic_ipv6_proxy - assert 500 == 200
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_assert_fingerprint_md5 - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_assert_hostname[localhost] - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_assert_hostname[127.0.0.1] - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_assert_hostname[::1] - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_assert_hostname_non_matching[::1] - assert "hostname \\'example.com\\' doesn\\'t match \\'0:0:0:0:0:0:0:1\\'" in '<ExceptionInfo MaxRetryError(\'HTTPSConnectionPool(host=\\\'localhost\\\', port=45835): Max retries...
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_ipv4_san - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_ipv6_san - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
FAILED test/with_dummyserver/test_proxy_poolmanager.py::TestHTTPSProxyVerification::test_https_proxy_no_san_hostname_checks_common_name - urllib3.exceptions.ProxySchemeUnsupported: TLS in TLS requires SSLContext.wrap_bio() which isn't available on non-native SSLContext
======================================================= 35 failed, 1743 passed, 244 skipped, 54 warnings in 157.51s (0:02:37) ========================================================

27 of them appear to be caused by a new CPython bug tracked in python/cpython#116764. Applying a patch from python/cpython#116801 reduces the number of errors to 8. To silence the errors we can add temporary xfail to the proxy tests when Python is 3.13.0a5.

The remaining failures of `test_client_intermediate` and `test_client_key_password`

============================================================================== short test summary info ===============================================================================
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=38853): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=38853): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=32965): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=32965): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=33931): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=33931): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=36791): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=36791): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
======================================================== 8 failed, 1770 passed, 244 skipped, 59 warnings in 133.11s (0:02:13) ========================================================

There is a new change related to certificates in 3.13.0a5 python/cpython#112389, unapplying it fixes the errors. In case of these 8 failures, it's likely that we need to adapt urllib3's tests to the striker SSL context flags.

[illia-v]

The remaining failures of test_client_intermediate and test_client_key_password

============================================================================== short test summary info ===============================================================================
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=38853): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=38853): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=32965): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=32965): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=33931): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_2::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=33931): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_intermediate - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=36791): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
FAILED test/with_dummyserver/test_https.py::TestHTTPS_TLSv1_3::test_client_key_password - urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=36791): Max retries exceeded with url: /certificate (Caused by ProtocolError('Connection aborted.', ...
======================================================== 8 failed, 1770 passed, 244 skipped, 59 warnings in 133.11s (0:02:13) ========================================================

There is a new change related to certificates in 3.13.0a5 python/cpython#112389, unapplying it fixes the errors. In case of these 8 failures, it's likely that we need to adapt urllib3's tests to the striker SSL context flags.

This is blocked by python-trio/trustme#642