Specific package from install environment in build environment.

[Tim-Salzmann]

Hello,

In the build process of my package (pip install), I link (CMake) against libraries by another package. Therefore, I either need to install the exact same package in the isolated build environment (1) or know the package path in the install environment and pass this information down to the isolated build environment (2).

The package in question I need to link against is PyTorch.

(1) Defining the same package version in build-system/requires and dependencies does not guarantee the same installed package. A GPU/Cuda version in the install environment will show as torch==2.0.0 but placing torch==2.0.0 in the build-system/requires will download and install torch CPU version in the build environment. Further, I do not want to restrict the user to either CPU or GPU version but rather want to use the same package in the build environment, which is already installed in the install environment.

(2) Determining the path of the package from the install environment is trivial:
python -c "import torch; print(torch.__file__)". However, I could not find a way to run this during the installation procedure in the install environment. Getting this information somewhere in setup.py would be sufficient to pass it down to the build system.

I would appreciate any help.

Thanks

[abravalheri]

Hi @Tim-Salzmann, I am afraid that the way the installation procedure is currently defined by PyPA decouples the building phase with the installation phase.

There is no guarantee the "installation environment" is "ready" (or even available at all) when running the "build" phase. This means that setuptools itself has no access to the "installation environment", and therefore even if we wanted, we have no means of providing references to it in setup.py.

The only alternative I can see if you want to go via the (2) route is explicitly asking for pip to disable build isolation (pip install --no-build-isolation ...). This option asks pip to use the same environment for building and installing the package. Of course, the downside is that pip relinquishes upon you the responsibility of installing the build dependencies into the environment, and it will fail if they are not available.

Regarding option (1):

A GPU/Cuda version in the install environment will show as torch==2.0.0 but placing torch==2.0.0 in the build-system/requires will download and install torch CPU version in the build environment.

Version identifiers should unequivocally identify an specific implementation of a package, except for the system platform, ABIs etc... But then, the wheel spec does not cover GPUs very well, so I guess PyTorch is using versions in a non-orthodox way/workaround.

Having a look on the PyTorch private index we can see things like torch-2.0.1+cu118-cp38-cp38-win_amd64.whl and torch-2.0.0+cpu.cxx11.abi-cp38-cp38-linux_x86_64.whl. Note how they have a local version segment (e.g. +cu118 or +cpu.cxx11.abi). So maybe you can explore this extra segment when trying to ensure the same package is installed?

Further, I do not want to restrict the user to either CPU or GPU version but rather want to use the same package in the build environment, which is already installed in the install environment.

Please note that this is not possible. There might be no such a thing as a version "already installed in the install environment". For example, imagine that the user is installing your package in a brand new environment. There is no PyTorch already there... instead it will be installed as a consequence of being defined as a dependency of your package. In this case the build environment comes first.

If you want the workflow to be exactly as you described, I am afraid you need to give specific instructions for your users to manually install your package. E.g.:

1. Manually install PyTorch
2. Manually install all build dependencies for your package
3. Install your package using --no-build-isolation

(which basically brings you back to the same comments for option (2))

---

If you want to avoid giving such detailed/manual install instructions to your users, I think that the best place to find advice is to ask to the PyTorch maintainers. They are the people that know about the intricacies of their workaround for wheel's lack of support for GPU tagging and how that interplays with 3rd-party packages/plugins.

[Tim-Salzmann]

Hi,

Thanks for the detailed reply! This is much appreciated!

Having a look on the PyTorch private index we can see things like torch-2.0.1+cu118-cp38-cp38-win_amd64.whl and torch-2.0.0+cpu.cxx11.abi-cp38-cp38-linux_x86_64.whl. Note how they have a local version segment (e.g. +cu118 or +cpu.cxx11.abi). So maybe you can explore this extra segment when trying to ensure the same package is installed?

This crossed my mind too. However, I could not find a way to determine the name of the installed wheel in the install environment during the setup routine. Any ideas?

In the meantime, I will opt for the easy solution.

• Manually install PyTorch
• Manually install all build dependencies for your package
• Install your package using --no-build-isolation

Thank you for the help!
Tim

[abravalheri]

However, I could not find a way to determine the name of the installed wheel in the install environment during the setup routine. Any ideas?

The name of the wheel is a bit complicated, but you could try importlib.metadata.version and yield torch==<version> as dependency.

But if the same version (including local segments) refers to 2 different artifacts depending on the package index (e.g. PyPI vs PyTorch custom index) that does not help a lot¹.

Footnotes

1. Roughly speaking, if there is no isomorphism between <package name> + <version identifier> and implementation (for a given platform architecture, OS, ...) there is very little the dependency resolution algorithm implemented in pip can do... So asking PyTorch itself how to get the correct dependency specifier would be important... ↩

[Tim-Salzmann]

Quick follow up:

• Manually install PyTorch
• Manually install all build dependencies for your package
• Install your package using --no-build-isolation

this works when run in a virtual environment. However when I try to install system-wide (happens in my github CI pipeline) I get the following output:

Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info
  writing /tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-0p1n1vg2/UNKNOWN.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml): started
  Running command Building wheel for UNKNOWN (pyproject.toml)
...

Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Successfully installed UNKNOWN-0.0.0

No dependencies are installed either and the package is not usable.
Could this be related to #3269

I appreciate any help!

[abravalheri]

@Tim-Salzmann, have you installed the latest versions of setuptools, wheel, cmake and any other build dependencies in your environment before proceeding with pip install --no-build-isolation ...? (remember to perform step 2 as discussed above: "Manually install all build dependencies for your package")