2023.5.7 Causing CI/CD issues

[AndyMoreland]

Good morning! I'm not sure if this is a PDM, requests, or certifi issue, but as of the latest release (~17 minutes ago now) I'm seeing my CI/CD builds which run

pdm install

fail with this error:

#20 32.13 in unpack_link
#20 32.13     resp = session.get(link.normalized, stream=True)
#20 32.13   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600,
#20 32.13 in get
#20 32.13   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587,
#20 32.13 in request
#20 32.13   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701,
#20 32.13 in send
#20 32.13   File "/usr/local/lib/python3.10/site-packages/cachecontrol/adapter.py", line 
#20 32.13 57, in send
#20 32.13     resp = super(CacheControlAdapter, self).send(request, **kw)
#20 32.13   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 459,
#20 32.13 in send
#20 32.13   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 262,
#20 32.13 in cert_verify
#20 32.13 OSError: Could not find a suitable TLS CA certificate bundle, invalid path: 
#20 32.13 /usr/local/lib/python3.10/site-packages/certifi/cacert.pem

Some relevant packages that are installed --

pdm-2.5.5-py3-none-any.whl
urllib3-1.26.15-py2.py3-none-any.whl
requests-2.29.0-py3-none-any.whl
certifi-2023.5.7-py3-none-any.whl

The timing for the start of these failures is to-the-minute with the latest release.

[Lukasa]

Interesting. If I install into a clean virtualenv I don't have this issue at all, which suggests the wheel is intact. I wonder if something is incompatible with pdm. Can you try a clean environment with pdm and one with pip to confirm whether they're behaving differently?

[hurricanefrog]

Not sure if this is related, but: I'm having problems with python-certifi-win32 (which depends on this package) in Gitlab Pipelines as well. Installing a package from a Python Package repository (GitLab, custom certificate, CA is in Windows trust store) via pip fails with

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate chain too long (_ssl.c:1129)'))

Downgrading to the previous certifi version fixes this

[riton]

Hi, I'm absolutely not sure that this is related, but I'm facing the "same" end-user message Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem

Here are the exact steps I'm using in my pipelines using python 3.9.16.

# pip3.9 install --upgrade pip wheel
# pip3.9 install poetry
# poetry self lock
# poetry self add 'poetry-dynamic-versioning[plugin]' <--- This command will fail

They were working until very recently.

While executing the last command, I see a message such as

  • Updating certifi (2023.5.7 /usr/local/lib/python3.9/site-packages -> 2023.5.7)

even if I already had certifi==2023.5.7 installed...

After that, the /usr/local/lib/python3.9/site-packages/certifi/cacert.pem has vanished.

Here are the full trace on a RedHat 8 based system: trace.txt

Hope this may help

[natali-avr]

Hey, I am having the same issue using PDM.
Does anyone found a solution for it?
Thanks

[sigmavirus24]

@natali-avr I have a theory about what could be causing this but I think everyone is running on some remote CI system without the ability to replicate what's happening locally.

I believe the issue at the core of @AndyMoreland and @riton's problems is that there is a global/system-level upgrade of certifi happening that is breaking things and it's related to the container/infrastructure things are running on. As a result, the best solution is to run PDM/poetry from within a virtual environment (as Cory hinted at). I think @hurricanefrog is having a different issue, though, so I don't know which of the errors included in this thread led you here.

[didib]

Hi all. I ran into a similar issue, and managed to fix it, hope this might help others.

I want to install some project that uses pdm, and pdm install fails with:

OSError: Could not find a suitable TLS CA certificate bundle, invalid path:
/foo/t/contcert/venv/lib64/python3.9/site-packages/certifi/cacert.pem

I am on RHEL 8, using virtualenv, created with 'python3.9 -m venv'.

It turned out that the project had pdf.lock saying:

[[package]]
name = "certifi"
version = "2023.7.22"

But with pip install pdf, I got:

$ pip list | grep certifi
certifi            2023.11.17

So when I ran pdm install, it removed certifi, perhaps in preparation for installing the correct/requested version, and then, some other code failed due to missing certifi (with the above error). The solution was to install the correct certifi before trying to let pdm do its job:

pip install certifi==2023.7.22
pdm install

One might claim that pdm should have done this by itself, instead of removing. I don't know pdm at all, so not opening there an issue (yet?).

[AvlWx2014]

Hi all. I ran into a similar issue, and managed to fix it, hope this might help others.

I want to install some project that uses pdm, and pdm install fails with:

OSError: Could not find a suitable TLS CA certificate bundle, invalid path:
/foo/t/contcert/venv/lib64/python3.9/site-packages/certifi/cacert.pem

I am on RHEL 8, using virtualenv, created with 'python3.9 -m venv'.

It turned out that the project had pdf.lock saying:

[[package]]
name = "certifi"
version = "2023.7.22"

But with pip install pdf, I got:

$ pip list | grep certifi
certifi            2023.11.17

So when I ran pdm install, it removed certifi, perhaps in preparation for installing the correct/requested version, and then, some other code failed due to missing certifi (with the above error). The solution was to install the correct certifi before trying to let pdm do its job:

pip install certifi==2023.7.22
pdm install

One might claim that pdm should have done this by itself, instead of removing. I don't know pdm at all, so not opening there an issue (yet?).

Just popping in to say that the root cause in this case was that PDM was being used to provision the environment it was itself installed in. The project in question had a dependency on an older version of certifi (2023.7.22) than PDM currently ships with (2023.11.17). When pdm install was run PDM started downgrading certifi breaking halfway through once the "uninstall" half of the downgrade operation had completed.

Not sure it helps with the original question, but in this case at least you can avoid the issue by one of:

• Install PDM using a system python interpreter with pip install pdm
• Install PDM as a user site package using a system Python interpreter with pip install --user pdm
• Install PDM in an isolated virtual environment maintained separately from any project environments. The downside is that PDM's venv would require activation before the command is available to you.