SSLCertVerificationError when behind a firewall — system SSL certs are not respected

[sm-Fifteen]

Every now and then (not sure what triggers it), when running a Prefect Orion server on Windows while behind a corporate VPN, I get strange SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)') errors from this line:

prefect/src/prefect/orion/services/telemetry.py

Lines 102 to 107 in 628b256

	async with httpx.AsyncClient() as client:
	result = await client.post(
	"https://sens-o-matic.prefect.io/",
	json=heartbeat,
	headers={"x-prefect-event": "prefect_server"},
	)

The error doesn't seem to affect the user-facing behavior of the application, but it certainly clutters the logs.

This is occuring because httpx uses certifi for its SSL validation, which is completely fine on normally configured machines, but a lot of corporate firewalls do SSL inspection, where all external traffic is intercepted and re-encrypted using a self-signed authority (so basically a man-in-the-middle on all employee workstations). That CA certificate is present in the device CA stores, but is naturally absent from stores like certifi, so httpx will reject all responses to external HTTP ressources.

The httpx doc says this can easily be fixed by using the system SSL context instead of the default one.

import ssl
import httpx
context = ssl.create_default_context()

async with httpx.AsyncClient(verify=context) as client:
        response = await client.get('https://example.org')

This also affects some example flows, like the "Basic Orchestration" github stars one, which also uses httpx.

[github-actions]

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

[zanieb]

We're willing to accept the addition of a setting that changes the default behavior as discussed in #7596 (comment)

[jawnsy]

Hey there! After further investigation, we believe that you can handle this situation by setting SSL_CERT_FILE or SSL_CERT_DIR as needed to use your own certificates. For more information, please see the httpx documentation for SSL_CERT_FILE.

We're therefore closing this issue. Please feel free to reply if this is insufficient for your needs.

[sm-Fifteen]

Actually, now that truststore is starting to become stable (it hit beta status earlier this month and is being integrated with pip), it should be possible to swap certifi with it and just do this on Python 3.10 and above:

import httpx
import ssl
import truststore

ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client = httpx.Client(verify=ssl_context)

It might end up being fixed upstream on httpx's side at some point. though.

We're therefore closing this issue. Please feel free to reply if this is insufficient for your needs.

The issue was never really a showstopper in my case, since like I mentionned in my original post, it mainly affects (or rather affected at the time, it's been a while since I last checked) outgoing telemetry, and httpx usage as portrayed in the tutorial. User flows can still disable HTTPS validation or use truststore explicitly if your Python version is compatible.

Note for bystanders: Using SSL_CERT_FILE on Windows is a bit of a hassle, since you need to use certlm.msc to dump the certificates (in base64 PEM encoding) from the system store and then concatenate the contents of these files (in order, starting from the root) manually to have the full trust chain in a single file. Once you have it, though, you can also use it with Pip (which uses REQUESTS_CA_BUNDLE) and Npm (which uses NODE_EXTRA_CA_CERTS).

[zanieb]

Thanks for the additional context!

Additionally, note that telemetry can always be disabled :)