Behavior change: v2.x raises SSLEOFError with Python3.10 where v2.x does not with Python3.8 #3100
Comments
|
maybe related to SSLEOFError in Python 3.10 not in Python 3.9 |
|
I knew that newer Python versions had issues with OpenSSL 3.0, but I'm surprised that just upgrading urllib3 to 2.0 makes a difference. I won't be able to help much without being able to reproduce, however. Is there any chance that you could capture a pcap file using Wireshark? |
|
I was trying to find a public server where I could reproduce the issue but had no luck. |
|
I used tcpdump to capture pcap files directly from within the Docker container which is connecting to the web server. Hope this helps identifying the difference.
For reference/comparison: working py38 urllib2 context
|
|
Thanks! This is great. I still need to analyze things more to understand the EOF error, but the server is using the Also, would you mind sharing a full trackeback instead of just the exception? |
|
A little bit of background: the server is similar to this demo from its manufacturer: http://netio-4c.netio-products.com/ I had to modify your script a bit because of the self-signed certificate (see below) but other than that, it is working. Unfortunately, I cannot control how my "frontend" (aka robotframework-requests) is utilizing Python's requests library which is utilizing urllib3 to make requests. I can only control whether to verify certificates (which I don't) and to ignore warnings. import ssl
from urllib3 import PoolManager
from urllib3.util import create_urllib3_context
from urllib3 import ssl
from urllib3 import make_headers
ctx = create_urllib3_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
ctx.load_default_certs()
ctx.set_ciphers("AES256-SHA256")
type = {'Content-Type': 'application/json'}
auth = make_headers(basic_auth=f'read:[read password]')
headers = { **type, **auth }
with PoolManager(ssl_context=ctx) as pool:
resp = pool.request("GET", "https://[IP]/netio.json", retries=0, headers=headers)
print(resp.data)EDIT (another experiment): root@595a566742be:/# python3 test.py
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 635, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 774, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 459, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 503, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:1007)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "//test.py", line 19, in <module>
resp = pool.request("GET", "https://[IP]/netio.json", retries=0, headers=headers)
File "/usr/local/lib/python3.10/dist-packages/urllib3/_request_methods.py", line 110, in request
return self.request_encode_url(
File "/usr/local/lib/python3.10/dist-packages/urllib3/_request_methods.py", line 143, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 433, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[IP]', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))But when I execute the script that throws the exception above (with the commented line regarding the additional cipher) with Python 3.8 instead of Python 3.10, there's no exception and the script works. EDIT again: here's the traceback from the original issue: 20230808 10:29:26.834 - FAIL - SSLError: HTTPSConnectionPool(host='10.1.1.32', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
20230808 10:29:26.836 - DEBUG - Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 635, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 774, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 459, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 503, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:1007)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 874, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 874, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 874, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.1.1.32', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/RequestsLibrary/utils.py", line 154, in decorator
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/RequestsLibrary/RequestsOnSessionKeywords.py", line 31, in get_on_session
response = self._common_request("get", session, url,
File "/usr/local/lib/python3.10/dist-packages/RequestsLibrary/RequestsKeywords.py", line 37, in _common_request
resp = method_function(
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 548, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 648, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.1.1.32', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)'))) |
No. The key used for a certificate is orthogonal to the cipher suites supported by the server for TLS. I would also expect any product you purchase to allow you to customize the supported cipher suites so that you could fix this there. |
|
Thanks for the clarification @sigmavirus24
Unfortunately, this is out of scope for the product used. The only configuration that is available, is to enable/disable https (including custom ports) and an upload for a custom certificate incl. its private key. Is the cipher AES256-SHA256 uncommon/weak/deprecated? Nevertheless, with Python 3.8 urllib3v2 behaves differently than with Python 3.10 which is something that should be at least understood (if it's acceptable or not). My work-around currently is not using urllib3v2 at all. |
|
Yes, we mentioned in the release notes that we would now be using the OpenSSL default cipher suite instead of the custom list in 1.26.x: see #2082. If you squint a bit, this is similar to a good cryptography practice: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number. What was surprising here is that you did not report an SSL handshake error, but an SSLEOFError. (Which is why I'd like to see the full traceback.) That said, you're the fourth user reporting us an issue with the OpenSSL cipher suite, so we will likely need to reevaluate it and add more ciphers to it. Other reports include: |
Do you need more than the traceback I provided a few comments back? |
|
I had indeed missed the edit, thanks this is what I needed! I'll tell you if I need more things as this exercise is very valuable. |
|
My bad. I didn't want to spread the traces throughout the thread, so I edit'ed several times... |
Yes, it is weak because RSA does not support PFS and CBC mode is vulnerable to plaintext attacks. See https://ciphersuite.info/cs/TLS_RSA_WITH_AES_256_CBC_SHA256/ for more explanations. But it's included by default on Ubuntu or Fedora, so your operating system seems to be configured differently. Indeed, when I make requests from my Fedora laptop, this cipher is included. And looking at https://ubuntu.com/server/docs/openssl, running How is your OpenSSL configured? It seems to be stricter than that, and you'll need to fix it.
Looking more closely at the packet capture with Wireshark, I now understand why you got
(I don't understand why Wireshark is reporting TLSv1 here, since urllib3 2.0 defaults to TLS 1.2 and the "Handshake Protocol Version" in Wireshark (not shown here) is TLS 1.2.) Anyway, you can see that the client sends a Client Hello TLS frame in packet 4. While the server receives that packet, it then close the TCP connection in packet 6, which the client acknowledges. It's a violation of the protocol, instead an
|
To rule out any side effects between OpenSSL versions (1.1.1f and 3.0.2) I ran my experiments from above in a plain ubuntu:focal (openssl 1.1.1f) and ubuntu:jammy (openssl 3.0.2) Docker container - both with Python 3.8 and 3.10.
I did not change any of openssl's default configuration bundled with the official Ubuntu Docker image. I only install Python and some Python libraries additionally. I can confirm that the output from
For what it's worth, here's the /etc/ssl/openssl.cnf from ubuntu:focal with and the /etc/ssl/openssl.cnf from ubuntu:jammy with |
|
Oh, wow, so using the following script on the Ubuntu Focal Docker image: from urllib3.util import create_urllib3_context
ctx = create_urllib3_context()
for cipher in ctx.get_ciphers():
print(cipher["description"])I get the following output, with corresponds to your Wireshark ciphers: But now, if I add Which includes the line of interest: On my Fedora laptop, I also see a difference where DEFAULT adds some CBC ciphers and removes some CCM ciphers. We can easily call urllib3/src/urllib3/util/ssl_.py Lines 297 to 300 in 53368df But I'd have to understand what is the difference between not calling Thanks for the responsiveness, it's very appreciated. |
|
Aha, if we don't set a default, we get the default from CPython, defined in https://github.com/python/cpython/blob/65ce3652fa47a34acf715ee07bf0a2fae2e0da51/Modules/_ssl.c#L151-L186 And indeed, with |
3 tasks
|
We continue getting reports about
|
|
It looks related with python/cpython#25574 . when python 3.9 handshake did not provide SSLEOFErrors or OSError when handshaking but python 3.10 provide SSLEOFErrors. it also related python/cpython#115627 but, i think it may have another issue on python ssl handshake. i will try to check it. |
|
Based on the wireshark packet, it seems that if it fails, the server ends the connection without any response. I reproduced the symptoms as follows. test_serverI used the "nc" command to end with ctrl+c without needing any response. nc -l -p 10000test_codeimport ssl
import socket
context = ssl.create_default_context()
with socket.create_connection(("localhost", 10000)) as sock :
with context.wrap_socket(sock , server_hostname="localhost") as ssock:
print(ssock.version())I ran this on a version-by-version basis and when the contents were updated on nc, I just pressed ctrl+c to terminate the connection. python:mainorange@32thread-server:~/project/python-contribute/keepworking/cpython/build$ ./python test.py
Traceback (most recent call last):
File "/home/orange/project/python-contribute/keepworking/cpython/build/test.py", line 6, in <module>
with context.wrap_socket(sock , server_hostname="localhost") as ssock:
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/orange/project/python-contribute/keepworking/cpython/Lib/ssl.py", line 455, in wrap_socket
return self.sslsocket_class._create(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
sock=sock,
^^^^^^^^^^
...<5 lines>...
session=session
^^^^^^^^^^^^^^^
)
^
File "/home/orange/project/python-contribute/keepworking/cpython/Lib/ssl.py", line 1077, in _create
self.do_handshake()
~~~~~~~~~~~~~~~~~^^
File "/home/orange/project/python-contribute/keepworking/cpython/Lib/ssl.py", line 1363, in do_handshake
self._sslobj.do_handshake()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1023)python 3.10orange@32thread-server:~/project/python-contribute/keepworking/cpython/build$ python3.10 test.py
Traceback (most recent call last):
File "/home/orange/project/python-contribute/keepworking/cpython/build/test.py", line 6, in <module>
with context.wrap_socket(sock , server_hostname="localhost") as ssock:
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1100, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)python 3.9orange@32thread-server:~/project/python-contribute/keepworking/cpython/build$ python3.9 test.py
Traceback (most recent call last):
File "/home/orange/project/python-contribute/keepworking/cpython/build/test.py", line 6, in <module>
with context.wrap_socket(sock , server_hostname="localhost") as ssock:
File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:1123)In my opinion, if the server terminates without responding to any warning, it is difficult for the client to display the correct information. as far as i know, providing information on negotiation failures is a non-essential recommendation in the TLS protocol because it can provide additional information to attackers. It is difficult to view EOF errors as a bug. urllib3 do not need any action for it. |
The protocol version of ClientHello's record layer is 1.0. It seems to be marked 1.0 in wireshark because negotiations were not made because the server did not respond to this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Subject
Setup: Using robotframework-requests library which in turn utilizes requests library to issue GET requests to a non-public web server over https. The web server is third-party.
The GET requests work fine using Python3.8, openssl [1.1.1f|3.0.2] and urllib3 2.x
The GET requests, however, raise following error when using the same context with only Python changed to 3.10:
SSLError: HTTPSConnectionPool(host='10.1.1.32', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))Environment
Reproduced on Ubuntu 20.04 LTS and 22.04 LTS (openssl 1.1.1f and openssl 3.0.2)
Combination of libs that raise the error (Ubuntu 22.04 LTS):
Combination of libs that do not raise the error (Ubuntu 22.04 LTS):
Difference between the two combinations:
pip install --upgrade --force-reinstall urllib3==1.26.15Combination of libs that do not raise the error as well (Ubuntu 20.04 LTS):
Steps to Reproduce
It's hard to reproduce publicly because the web server is not public.
But my findings lead to the SSL_OP_IGNORE_UNEXPECTED_EOF flag.
Expected Behavior
v2.x shall behave the same way with Python 3.10 as it does with Python 3.8 regarding the error described above (i. e. not raising any SSLEOFError)
Actual Behavior
v2.x in combination with Python 3.10 raises SSLEOFError
v2.x in combination with Python 3.8 does not raise SSLEOFError
The text was updated successfully, but these errors were encountered: