Pin version of packaging and other essential dependencies in PDM

[lambda]

Is your feature request related to a problem? Please describe.

Packaging 22.0 introduced a number of backwards incompatible changes, which can break projects using PDM:

• PDM fails to resolve on packages which have old versions with legacy version identifiers  #1554
• Issues with resolving dependency with python 3.11.1 #1556
• 22.0: The new requirement parser doesn't handle unnormalized specifiers pypa/packaging#629

In order to avoid problems with PDM updates breaking my CI pipelines, I have switched to installing PDM as pip install pdm~=2.3.1, in order to get bugfix releases but manually opt in to feature releases.

However, PDM doesn't lock its own dependencies, so a major release of Packaging 22.0 introduced several incompatible changes, without any update to the PDM version.

Describe the solution you'd like

PDM should lock the exact versions of its own dependencies, so that installing the same version of PDM gives the same results each time. Major version updates of dependencies that affect the PDM interface in backwards-incompatible ways, such as packaging, should at least only be done on feature releases of PDM.

[pawamoy]

I disagree. It's true that PDM is more an app than a library, but pinning dependencies brings more problems than it solves. I recently started developing a PDM plugin, so my project depends on PDM, and I definitely do not want to be restricted by pins in PDM's dependencies.

Packages incompatibilities can always be solved on the final user side, using constraints for example. On the contrary, locking dependencies gives no alternative to downstream users.

Just my opinion!

[lambda]

@pawamoy The whole reason we use PDM is so that we can have reproducible builds, without having unexpected dependency updates breaking us, while still being able to iterate quickly and update dependencies when we need to.

With PDM itself not having pinned dependencies, we lose that ability; updates to PDM or its dependencies can make our builds fail when there have been no changes on our side.

Of course, having proper support for lockfiles in the Python packaging ecosystem would address this, in which you could have a locked set of dependencies for usage as an application, while loose dependencies when used as a library, just as PDM allows us to do for packages managed by PDM. Unfortunately, there's a bootstrapping problem when installing PDM itself.

One possible way to work around this would be to have two published PDM packages; one with pinned dependencies for use as an application, and one with loose dependencies for use as a library.

[pradyunsg]

Another option is to have an optional extra in PDM that provides the pins (pdm[pinned]).

[frostming]

Agree with @pawamoy you can easily exclude a broken dependency version by pip install pdm packaging>=20,!=22.0

Another option is to have an optional extra in PDM that provides the pins (pdm[pinned]).

I don't think it is a good idea, this will increase the overall burden for the maintainers. They must update the dependencies very carefully and do full tests before publishing. Every broken change of any package may require them to upload a new release of the pinned version.

[frostming]

Also, note that #1554 and #1556 are different from pypa/packaging#629, they are not regression issues but breaking changes that remove deprecated behaviors.

As a library keeps evolving it will introduce breaking changes in an expected and well-noticed manner. If you are still relying on old behaviors you have to use older versions.

[franzhaas]

In my environment, some tools are installed using pipx, and one of those is PDM.

The occasional breakage due to an incompatible change in a library is not the most pleasant experience. But I can live with it. That each setup has a slightly different set of installed dependencies is what I want to avoid using PDM.

Is there an equivalent to

pipx install xxxx


where a pdm.lock file can be provided so that every installation has the same set of dependencies inside my organization?

[pawamoy]

You can run pip commands in a pipx env, for example to provide a constraints file to restrict or pin PDM's dependencies: pipx runpip pdm install pdm -c constraints.txt (the first pdm is the pipx venv name, then install ... are the arguments to pip).

[frostming]

You can run pip commands in a pipx env, for example to provide a constraints file to restrict or pin PDM's dependencies: pipx runpip pdm install pdm -c constraints.txt (the first pdm is the pipx venv name, then install ... are the arguments to pip).

Or simply pipx install --pip-args="-c constraints.txt" pdm will also work. As @pawamoy and I stated, that is easy to solve on the user side.

[franzhaas]

I tried to avoid having a slighlty different pdm installation on every machine by using pdm-packer.

After some investigation I believe that pdm is not suitable to be packed as a zip app, due its usage of file at various places right now.

@models.environment.py i believe there it is no problem.
@models.specifiers.py I believe it is hard to not break the API to provide the MAX_VERSIONS_FILE API without usage of file

there are some more places where file is used, but I did stop investigating at this point.

Question.:
Is PDM intended to be used as a zip app? I don't expect the maintainers to create the zip app, but to do it myself with pdm packer.

I am considering either going towards PDM as a zip app, pipx with requirements.txt, or PDM deployed with a windows installer... I did not make up my mind yet, but I definitely do not want to work against PDMs project goals.

[frostming]

Is PDM intended to be used as a zip app?

Not the case now, it won't work in a zipapp with the __file__ usages. No user has shown interest in creating zipapp for it. But it isn't hard to support, I will create a patch for that, Thanks.

[frostming]

Closed by 2.3.3