Use default configured ciphers instead of maintaining our own cipher list

[sethmlarson]

So originally I was planning on dropping more than just RSA for v2, however after speaking with @hynek it seems a lot of these ciphers are likely still in use and secure so we'll keep them.

However because there's less being removed here I'm wondering if we need the second part of #1989: the fallback to previous DEFAULT_CIPHERS if a deprecated TLS version is used.

cc @tiran

Closes #1989
Closes #2090

[codecov]

Codecov Report

Merging #2082 (0d5030c) into master (fe09c7f) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master     #2082      +/-   ##
===========================================
+ Coverage   99.90%   100.00%   +0.09%     
===========================================
  Files          25        25              
  Lines        2215      2216       +1     
===========================================
+ Hits         2213      2216       +3     
+ Misses          2         0       -2     

Impacted Files	Coverage Δ
src/urllib3/util/ssl_.py	100.00% <100.00%> (ø)
src/urllib3/__init__.py	100.00% <0.00%> (ø)
src/urllib3/connection.py	100.00% <0.00%> (ø)
src/urllib3/exceptions.py	100.00% <0.00%> (ø)
src/urllib3/contrib/socks.py	100.00% <0.00%> (ø)
src/urllib3/connectionpool.py	100.00% <0.00%> (ø)
src/urllib3/response.py	100.00% <0.00%> (+0.51%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe09c7f...0d5030c. Read the comment docs.

[pquentin]

Our cipher suite is pretty different from what @hynek recommends in https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/. Green lines with a + are what we add, and red lines with - are what we remove:

+ECDHE+AESGCM
+ECDHE+CHACHA20
+DHE+AESGCM
+DHE+CHACHA20
 ECDH+AESGCM
-ECDH+CHACHA20
 DH+AESGCM
-ECDH+AES256
-DH+AES256
-ECDH+AES128
+ECDH+AES
 DH+AES
-RSA+AESGCM
-RSA+AES
 !aNULL
+!eNULL
 !MD5
 !DSS
-!AESCCM;

I know we discussed this in #1496 (comment), but this pull request is the occasion to see if the assumptions we made then still hold.

[hynek]

ECDH and DH are technically static versions of ECDHE/DHE which means no PSF which means Very Bad™.

However:

1. No server I'm aware of ever supported them.
2. Older OpenSSL versions didn't allow to specifically ask for ECDHE/DHE. IIRC, it was actually a flag you had to set on top of your SSL context after asking for ECDH/DH but the details elude me.

So it could be a compatibility problem to ask for ECDHE, but I can't tell you how big of one. I just did a quick test with Docker, and Ubuntu 12.04 LTS (not supported anymore) with openssl 1.0.1f returns no ciphers if you ask for ECDHE:

root@52d0d1a855da:/# openssl version
OpenSSL 1.0.1f 6 Jan 2014
root@52d0d1a855da:/# openssl ciphers -v 'ECDHE+AESGCM'
Error in cipher list
140120603170464:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl_lib.c:1314:
root@52d0d1a855da:/# openssl ciphers -v 'ECDH+AESGCM'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD

Ubuntu 16.04 LTS comes with 1.0.2g and it works:

root@8bc8777688aa:/# openssl version
OpenSSL 1.0.2g  1 Mar 2016
root@8bc8777688aa:/# openssl ciphers -v 'ECDHE+AESGCM'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD

So the change happened somewhere between January 2014 and March 2016. :) It depends really how compatible you want to be and AFAIK has no security implications.

---

I'll add an explanation to the blog post.

[hynek]

(oh and I don't think DHE+AESGCM and DHE+CHACHA20 are a thing in the real world. If your server is modern enough for AES-GCM/ChaCha20, it most certainly is ready for ECDHE but I'm happy to be corrected here)

[tiran]

I have a more radical opinion: Don't touch the default cipher suites at all and just use the default settings.

Any platform with OpenSSL 1.1.1 should have a reasonable secure default settings for ciphers suites. In case the defaults are not secure, the issue should be solved on a platform level. Any override conflicts with system-wide settings, too. An admin may disable or enable ciphers in a system-wide config file. You could disable a required algorithm or enable an algorithm should be blocked.

Secure cipher suites settings were necessary with OpenSSL 0.9.8 and 1.0.1. Nowadays it's more or less cargo cult.

[sethmlarson]

@tiran I like this idea, and I see that you're helpfully tracking ciphers in your distro-truststore repository.

Is there a way that we can detect if OpenSSL has been configured with specific ciphers? I'd like to still use our list if OpenSSL is unconfigured to avoid issues like a cipher without encryption getting selected (or is this not a worry?)

Maybe we can also make this decision based on OpenSSL version?

[tiran]

Any OpenSSL 1.1.1 version is fine. Older versions are potentially problematic because they are no longer maintained by upstream. The default cipher suites in OpenSSL 1.1.1 are ok. The list contains some unnecessary PSK and SRP ciphers. Since urllib3 does not support pre-shared key and secure remote password login, you can safely ignore the extra ciphers.

Here is a list of ciphers with from a self-compiled OpenSSL 1.1.1h build without openssl.cnf loaded:

$ OPENSSL_CONF=/dev/zero bin/openssl ciphers -v
TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any      Au=any  Enc=CHACHA20/POLY1305(256) Mac=AEAD
TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
DHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=DH       Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES256-SHA  TLSv1 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
ECDHE-RSA-AES256-SHA    TLSv1 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES128-SHA  TLSv1 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA1
ECDHE-RSA-AES128-SHA    TLSv1 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
RSA-PSK-AES256-GCM-SHA384 TLSv1.2 Kx=RSAPSK   Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-PSK-AES256-GCM-SHA384 TLSv1.2 Kx=DHEPSK   Au=PSK  Enc=AESGCM(256) Mac=AEAD
RSA-PSK-CHACHA20-POLY1305 TLSv1.2 Kx=RSAPSK   Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
DHE-PSK-CHACHA20-POLY1305 TLSv1.2 Kx=DHEPSK   Au=PSK  Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-PSK-CHACHA20-POLY1305 TLSv1.2 Kx=ECDHEPSK Au=PSK  Enc=CHACHA20/POLY1305(256) Mac=AEAD
AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
PSK-AES256-GCM-SHA384   TLSv1.2 Kx=PSK      Au=PSK  Enc=AESGCM(256) Mac=AEAD
PSK-CHACHA20-POLY1305   TLSv1.2 Kx=PSK      Au=PSK  Enc=CHACHA20/POLY1305(256) Mac=AEAD
RSA-PSK-AES128-GCM-SHA256 TLSv1.2 Kx=RSAPSK   Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-PSK-AES128-GCM-SHA256 TLSv1.2 Kx=DHEPSK   Au=PSK  Enc=AESGCM(128) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
PSK-AES128-GCM-SHA256   TLSv1.2 Kx=PSK      Au=PSK  Enc=AESGCM(128) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-PSK-AES256-CBC-SHA384 TLSv1 Kx=ECDHEPSK Au=PSK  Enc=AES(256)  Mac=SHA384
ECDHE-PSK-AES256-CBC-SHA TLSv1 Kx=ECDHEPSK Au=PSK  Enc=AES(256)  Mac=SHA1
SRP-RSA-AES-256-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(256)  Mac=SHA1
SRP-AES-256-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(256)  Mac=SHA1
RSA-PSK-AES256-CBC-SHA384 TLSv1 Kx=RSAPSK   Au=RSA  Enc=AES(256)  Mac=SHA384
DHE-PSK-AES256-CBC-SHA384 TLSv1 Kx=DHEPSK   Au=PSK  Enc=AES(256)  Mac=SHA384
RSA-PSK-AES256-CBC-SHA  SSLv3 Kx=RSAPSK   Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-PSK-AES256-CBC-SHA  SSLv3 Kx=DHEPSK   Au=PSK  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
PSK-AES256-CBC-SHA384   TLSv1 Kx=PSK      Au=PSK  Enc=AES(256)  Mac=SHA384
PSK-AES256-CBC-SHA      SSLv3 Kx=PSK      Au=PSK  Enc=AES(256)  Mac=SHA1
ECDHE-PSK-AES128-CBC-SHA256 TLSv1 Kx=ECDHEPSK Au=PSK  Enc=AES(128)  Mac=SHA256
ECDHE-PSK-AES128-CBC-SHA TLSv1 Kx=ECDHEPSK Au=PSK  Enc=AES(128)  Mac=SHA1
SRP-RSA-AES-128-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=AES(128)  Mac=SHA1
SRP-AES-128-CBC-SHA     SSLv3 Kx=SRP      Au=SRP  Enc=AES(128)  Mac=SHA1
RSA-PSK-AES128-CBC-SHA256 TLSv1 Kx=RSAPSK   Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-PSK-AES128-CBC-SHA256 TLSv1 Kx=DHEPSK   Au=PSK  Enc=AES(128)  Mac=SHA256
RSA-PSK-AES128-CBC-SHA  SSLv3 Kx=RSAPSK   Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-PSK-AES128-CBC-SHA  SSLv3 Kx=DHEPSK   Au=PSK  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
PSK-AES128-CBC-SHA256   TLSv1 Kx=PSK      Au=PSK  Enc=AES(128)  Mac=SHA256
PSK-AES128-CBC-SHA      SSLv3 Kx=PSK      Au=PSK  Enc=AES(128)  Mac=SHA1

[sethmlarson]

@tiran Okay so we should detect OpenSSL 1.1.1 or later, if detected and no manual ssl_ciphers is given we use system defaults. If 1.1.0 or earlier we use our list sans-RSA. This matches what you're suggesting?

[pquentin]

Awesome, I like the new commit!

It's now much less important, but should we still try harder to match Hynek's list? Especially given we know it can only be used with versions earliers than 1.1.1

[sethmlarson]

For now we always want to set_ciphers() on SecureTransport pending investigation in #2091

[sethmlarson]

in #2091 we decided that SecureTransport ciphers were fine, so we can remove a lot of code now!

[sethmlarson]

We want to run all these tests on pyOpenSSL as well so we want to make this importable.

[sethmlarson]

@pquentin This PR now aligns the pyOpenSSL SSLContext behavior to be the same as the stdlib SSLContext.

I also fix up the SecureTransport cipher list to remove RSA and add one missing TLS 1.3 cipher (unsure if it'll really have an effect because SecureTransport doesn't support TLS 1.3 currently but oh well?)

[PleasantMachine9]

Shouldn't the PR title clarify that this is dropping non-forward-secrecy cipher suites instead?

There are plenty of cipher suites in the list that still include RSA, just not for KEX.
Or maybe something like Drop RSA key-exchange ciphers from defaults ?

[sethmlarson]

Okay, so I ran openssl ciphers on the latest Debian docker container and received the following ciphers:

TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
DHE-RSA-AES256-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256
DHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES256-SHA
ECDHE-RSA-AES256-SHA
DHE-RSA-AES256-SHA
ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA
DHE-RSA-AES128-SHA
AES256-GCM-SHA384
AES128-GCM-SHA256
AES256-SHA256
AES128-SHA256
AES256-SHA
AES128-SHA

Debian appears to be configured with DEFAULT@SECLEVEL=2.

then ran urllib3's cipher list (including RSA which I have put back) and received:

TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES256-CCM8
ECDHE-ECDSA-AES256-CCM
ECDHE-ECDSA-AES128-CCM8
ECDHE-ECDSA-AES128-CCM
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES256-SHA
ECDHE-RSA-AES256-SHA
ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA
DHE-RSA-AES256-CCM8
DHE-RSA-AES256-CCM
DHE-RSA-AES128-CCM8
DHE-RSA-AES128-CCM
DHE-RSA-AES256-SHA256
DHE-RSA-AES128-SHA256
DHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA
AES256-GCM-SHA384
AES128-GCM-SHA256
AES256-CCM8
AES256-CCM
AES128-CCM8
AES128-CCM
AES256-SHA256
AES128-SHA256
AES256-SHA
AES128-SHA

The only difference being that urllib3 includes CCM ciphers. Since Hynek recommends disabling CCM for being "safe but uncommon and slow" we'll do the same to align with Debian.

[pquentin]

That sounds great!