Windows symlinks no longer built into wheel for v2.3

[AlexanderWells-diamond]

Description

We have a repo with submodules, and use symlinks to bring files into the main Python module source.

As of version 2.3 the symlinked directory is no longer copied into the built wheel on Windows (it is for MacOS and Ubuntu builds). Rolling back to version 2.2.2 and the symlinked directory is copied, with no other configuration changes required.

The build log linked shows the Windows tests failing start because they cannot find a file that should be included by a symlink. The Ubuntu and MacOS tests find the file and run tests.

The symlink file itself can be seen here: https://github.com/dls-controls/pythonSoftIOC/blob/sort_setting_values/softioc/iocStatsDb

Build log

https://github.com/dls-controls/pythonSoftIOC/runs/4517544616?check_suite_focus=true

CI config

https://github.com/dls-controls/pythonSoftIOC/blob/7f95e6b885fe36315ed60c4f583de746068ab6fa/.github/workflows/code.yml#L69

[henryiii]

My guess this is due to pip 21.3, which no longer makes a complete copy of the source directory. The old behavior must have been resolving the symlinks during the copy process, causing it to happen to work. The new behavior just seems the symlinks as they are, and they are not handled. You'll probably need to override the setuptools bdist_wheel and maybe sdist commands.

You could force the old behavior for a little longer with PIP_USE_DEPRECATED=out-of-tree-build, except there's a bug (fixed in master) in cibuildwheel 2.3.0 that doesn't let you set this.

[henryiii]

Version 2.3.1 now allows PIP_USE_DEPRECATED=out-of-tree-build to be set in the cibw environment. You could use that to see if you can upgrade (then I'd work on fixing your setup.py so you'll support building in the future when this option is dropped from pip, or support for other build frontons like build).

[henryiii]

Example: pybind/cmake_example#67

[AlexanderWells-diamond]

Thank you for taking a look at this. However it seems that this fix doesn't work completely, at least not across all platforms and python versions we test against.

I ended up specifying PIP_USE_DEPRECATED: out-of-tree-build directly in the GitHub YAML config file, rather than in setup.cfg, as we don't use the cibuildwheel action. Ubuntu worked for Python 3.6 through 3.9, and Windows and MacOS on Python 3.6 and 3.7 work, but on 3.8 and 3.9 both platforms saw this error:

  + python -m ensurepip
  Looking in links: c:\Users\RUNNER~1\AppData\Local\Temp\tmp0oeln7iw
  Requirement already satisfied: setuptools in c:\cibw\python\python.3.8.10\tools\lib\site-packages (56.0.0)
  Requirement already satisfied: pip in c:\cibw\python\python.3.8.10\tools\lib\site-packages (21.1.1)
  + python -m pip install --force-reinstall pip -c C:\hostedtoolcache\windows\Python\3.7.9\x64\Lib\site-packages\cibuildwheel\resources\constraints-python38.txt
  An error occurred during configuration: option use-deprecated: invalid choice: 'out-of-tree-build' (choose from 'legacy-resolver')
Error: Command ['python', '-m', 'pip', 'install', '--force-reinstall', 'pip', '-c', 'C:\\hostedtoolcache\\windows\\Python\\3.7.9\\x64\\Lib\\site-packages\\cibuildwheel\\resources\\constraints-python38.txt'] failed with code 3. None

So if I'm understanding this right, the version of pip used in Python3.8 & 3.9 has already deprecated this feature and so this does not work (for us) as a workaround?

Currently pinning 2.2.2 will work, and it looks like we'll need to do some work in future as you suggest to change our build process to accommodate this.

[joerick]

Ah, perhaps you're specifying PIP_USE_DEPRECATED=out-of-tree-build for the whole cibuildwheel process? It will work better if you put that in the CIBW_ENVIRONMENT option e.g. CIBW_ENVIRONMENT: PIP_USE_DEPRECATED=out-of-tree-build - that way, it won't affect cibuildwheel's environment setup.

[AlexanderWells-diamond]

Trying it that way and it passes on every platform. Thank you, that'll do for now.

[henryiii]

as we don't use the cibuildwheel action

FYI, toml config does not have anything to do with the action - it works any way you call cibuildwheel. In fact, one of the major benefits of toml config is that it makes it easier to run it locally or change CI systems.