Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Deprecation warning when building wheels #2847

Closed
1 task done
domdfcoding opened this issue Nov 3, 2021 · 29 comments
Closed
1 task done

[BUG] Deprecation warning when building wheels #2847

domdfcoding opened this issue Nov 3, 2021 · 29 comments
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.

Comments

@domdfcoding
Copy link
Contributor

domdfcoding commented Nov 3, 2021

setuptools version

setuptools==58.3.0 or setuptools==58.4.0

Python version

Python 3.8

OS

Ubuntu 20.04

Additional environment information

No response

Description

Attempting to install or build a wheel of a project using setuptools results in the following warning:

venv/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
Full traceback:
  Traceback (most recent call last):
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 20, in <module>
      setup(
    File "venv/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "venv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 301, in run
      install = self.reinitialize_command('install',
    File "/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 214, in reinitialize_command
      cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
    File "/usr/lib/python3.8/distutils/cmd.py", line 305, in reinitialize_command
      return self.distribution.reinitialize_command(command,
    File "/usr/lib/python3.8/distutils/dist.py", line 938, in reinitialize_command
      command = self.get_command_obj(command_name)
    File "/usr/lib/python3.8/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "venv/lib/python3.8/site-packages/setuptools/__init__.py", line 178, in __init__
      _Command.__init__(self, dist)
    File "/usr/lib/python3.8/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/venv/lib/python3.8/site-packages/setuptools/command/install.py", line 34, in initialize_options
      warnings.warn(
  setuptools._deprecation_warning.SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

This occurs with both build and pip (pip install and pip wheel).

Expected behavior

No warning is emitted when building a wheel or installing using pip or build.

How to Reproduce

  1. Install setuptools 58.3.0 or 58.4.0
  2. Run PYTHONWARNINGS=error pip wheel domdf_python_tools==3.1.0 --no-binary domdf_python_tools -v --no-deps

This also occurs with whey and natsort.

Output

Looking in indexes: http://localhost:3141/root/staging
Collecting domdf_python_tools==3.1.0
  Downloading http://localhost:3141/root/pypi/%2Bf/c55/b356a87d53841/domdf-python-tools-3.1.0.tar.gz (99 kB)
  Installing build dependencies: started
  Running command /tmp/venv/bin/python3 /tmp/pip-standalone-pip-7m38gblg/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-myy0rzw0/overlay --no-warn-script-location --no-binary domdf-python-tools --only-binary :none: -i http://localhost:3141/root/staging -- 'setuptools>=40.6.0' 'wheel>=0.34.2'
  Looking in indexes: http://localhost:3141/root/staging
  Collecting setuptools>=40.6.0
    Downloading http://localhost:3141/root/pypi/%2Bf/e8b/1d3127a0441fb/setuptools-58.4.0-py3-none-any.whl (946 kB)
  Collecting wheel>=0.34.2
    Downloading http://localhost:3141/root/pypi/%2Bf/210/14b2bd93c6d00/wheel-0.37.0-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel, setuptools
  Successfully installed setuptools-58.4.0 wheel-0.37.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp32ew7q_j
  running egg_info
  writing domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to domdf_python_tools.egg-info/requires.txt
  writing top-level names to domdf_python_tools.egg-info/top_level.txt
  reading manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpenyjg9hk
  running dist_info
  creating /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info
  writing /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: domdf-python-tools
  Building wheel for domdf-python-tools (pyproject.toml): started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp_1m0sf3w
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/domdf_python_tools
  copying domdf_python_tools/bases.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/paths.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/secrets.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/dates.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/delegators.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/import_tools.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/versions.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/iterative.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/utils.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/__init__.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/terminal.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/typing.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/words.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/doctools.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/stringlist.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/_is_match.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/pretty_print.py -> build/lib/domdf_python_tools
  creating build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/sizes.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/utils.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/classes.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/__init__.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/units.py -> build/lib/domdf_python_tools/pagesizes
  creating build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_metadata.py -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/__init__.py -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_resources.py -> build/lib/domdf_python_tools/compat
  running egg_info
  creating domdf_python_tools.egg-info
  writing domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to domdf_python_tools.egg-info/requires.txt
  writing top-level names to domdf_python_tools.egg-info/top_level.txt
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  copying domdf_python_tools/google-10000-english-no-swears.txt -> build/lib/domdf_python_tools
  copying domdf_python_tools/py.typed -> build/lib/domdf_python_tools
  copying domdf_python_tools/compat/importlib_metadata.pyi -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_resources.pyi -> build/lib/domdf_python_tools/compat
  Traceback (most recent call last):
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 20, in <module>
      setup(
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 301, in run
      install = self.reinitialize_command('install',
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 214, in reinitialize_command
      cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
    File "/usr/lib/python3.8/distutils/cmd.py", line 305, in reinitialize_command
      return self.distribution.reinitialize_command(command,
    File "/usr/lib/python3.8/distutils/dist.py", line 938, in reinitialize_command
      command = self.get_command_obj(command_name)
    File "/usr/lib/python3.8/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 178, in __init__
      _Command.__init__(self, dist)
    File "/usr/lib/python3.8/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/command/install.py", line 34, in initialize_options
      warnings.warn(
  setuptools._deprecation_warning.SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  Building wheel for domdf-python-tools (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for domdf-python-tools
Failed to build domdf-python-tools
ERROR: Failed to build one or more wheels

Code of Conduct

  • I agree to follow the PSF Code of Conduct
@domdfcoding domdfcoding added bug Needs Triage Issues that need to be evaluated for severity and status. labels Nov 3, 2021
@jaraco
Copy link
Member

jaraco commented Nov 3, 2021

Agreed this should be suppressed when building wheels.

@jaraco jaraco closed this as completed in fad059e Nov 3, 2021
@mara004
Copy link

mara004 commented Feb 13, 2022

I am still experiencing this warning with setuptools 60.8.2 when using bdist_wheel.
(The project in question is pypdfium2 - just clone the repository and call pip3 install . -v to reproduce)

@abravalheri
Copy link
Contributor

abravalheri commented Feb 13, 2022

@mara004, this seems to be related to the way pip is working internally.

I recon the project needs to adhere to PEP 517 and PEP 518, i.e. a pyproject.toml file needs to be present in the root of the project with contents similar to the following:

# pyproject.toml
[build-system]
requires = ["setuptools>=46.1.0", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"

pypdfium2 in particular seems to depend on the __legacy__ behaviour, and also explicitly require the wheel dependency to run.

(I also recommend updating pip version).

@mara004
Copy link

mara004 commented Feb 13, 2022

@abravalheri First of all, thanks for the response.

(I also recommend updating pip version).

I am already running the latest pip version 22.0.3

i.e. a pyproject.toml file needs to be present in the root of the project

Sorry, adding the above pyproject.toml breaks the build with obscure errors. Anyway, I don't quite understand the background - why should setup.cfg not be sufficient?

pypdfium2 in particular seems to depend on the __legacy__ behaviour

What would I have to change so that the project does not rely on legacy behaviour?

@abravalheri
Copy link
Contributor

Hi @mara004,

Sorry, adding the above pyproject.toml breaks the build with obscure errors. Anyway, I don't quite understand the background - why should setup.cfg not be sufficient?

The thing pyproject.toml would do is to change the way pip is building the project.
It seems to me that pip tries to run python setup.py when it does not find pyproject.toml.
When pyproject.toml exists, it will run the functions in setuptools.build_meta instead.

Recently, setuptools reserved to itself the capacity of executing setup.py files directly ("demoted" it to an internal API), for the reasons explained in this article. So the warning is a heads up to developers saying that things might change in the future.

What would I have to change so that the project does not rely on legacy behaviour?

The main difference between the legacy and the standard backend is that __legacy__ automatically adds the directory that contains the setup.py script to sys.path. The standard backend does not do that.

@mara004
Copy link

mara004 commented Feb 13, 2022

Thanks for the help, this is really appreciated.
I'll have to look into pyproject.toml and the related PEPs more thoroughly to see if I can change my setup infrastructure. This will be quite complex since pypdfium2 has multiple setup files for different platforms, and it's totally built around some peculiarities of setup.py. (The project uses external binaries, which is a thing that no other Python library I know of does.)

@abravalheri
Copy link
Contributor

abravalheri commented Feb 13, 2022

@mara004, there are 3 things to consider in the case of pypdfium.

  • The setup.py script is importing from the platform_setup folder as if it was a package. However, it does not contain a __init__.py file.

  • The sdist does not seem to contain the platform_setup folder.

    • The installers are very likely to consider the sdist as an intermediate step to obtain the wheel. So they will create the sdist first and then create the wheel from it, not from the original code.

    • Since the platform_setup is not part of the final package, setuptools will not include it by default in the sdist. You need to write your own MANIFEST.in or simply use a plugin such as setuptools-scm, for example:

      # pyproject.toml
      [build-system]
      requires = ["setuptools>=46.1.0", "setuptools-scm", "wheel"]
      build-backend = "setuptools.build_meta:__legacy__"
  • If you don't want to use __legacy__, you either have to keep all your code inside setup.py or find a way to signal to Python's import system where to find the auxiliary packages you are importing in setup.py.

@mara004

This comment was marked as outdated.

@mara004
Copy link

mara004 commented Feb 15, 2022

I've now had some more time to look into this, and unfortunately it seems like it is simply not possible to do what I need with a PEP 517 compliant build system.

@abravalheri Re-reading your previous comment, I think you might not yet have understood what my setup code really does, so I'll try to explain the project's requirements and the current approach:

  • The wheels I package for PyPI need to include pre-built binaries. It must be possible to build all wheels on a single platform to avoid being dependent on CIs or virtualisation.
  • This is implemented with platform-specific setup files that each call a shared base setup function with the corresponding platform name as argument. Wheels for each platform can then be built using setup_{platform-name}.py bdist_wheel.
  • There is another setup file to create the source distribution (sdist). This will simply package the content of the repository into a tarball, without moving in binaries/bindings.
  • The main setup.py file checks whether pre-built binaries are available for the host platform. If so, it downloads them, creates the bindings, and calls the corresponding platform setup function. Otherwise, a source build will be attempted.

This has the following advantages:

  • When installing from PyPI, users of platforms with pre-built binaries will get finished wheels, otherwise they get the sdist and PDFium will be built from source.
  • setup.py also detects the fastest installation method, so it is possible to quickly install from git main and test changes.
  • It is very simple to make a release.

The approach of PEP 517 and the deprecation of the setup.py commands completely break this strategy:

  • There can only be a single setup file that is set as an entry point in pyproject.toml. One cannot call different setup files on an individual basis anymore.
  • Consequently, cross-packaging is impossible to do, though it would technically be the easiest solution for this case.

A potential fix to allow cross-packaging again would be to add a --setup-file argument or similar to the python3 -m build command, so that one can point the wheel builder at custom setup code to use.


Since the platform_setup is not part of the final package, setuptools will not include it by default in the sdist. You need to write your own MANIFEST.in or simply use a plugin such as setuptools-scm

From what I have seen, sdist should automatically take all files in the repository. When I run python3 platform_setup/setup_sdist.py sdist, the platform_setup folder and everything else is included as expected, without the need for additional configuration. Furthermore, MANIFEST.in is only relevant for packaging wheels, not source distributions.
Unless PEP 517 should have changed this, your statement seems to be incorrect, or have I overlooked anything?

@abravalheri
Copy link
Contributor

Hi @mara004, please find my comments bellow:

From what I have seen, sdist should automatically take all files in the repository.

When I run:

rm -rf dist build src/*.egg-info
python setup.py sdist
tar tf dist/pypdfium2-0.12.0.tar.gz

I cannot find the platform_setup folder inside the distribution.
When moving towards PEP 517, python setup.py sdist is what effectively runs, so I believe it will produce an sdist without the required files.

Furthermore, MANIFEST.in is only relevant for packaging wheels

This statement is incorrect, please check https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#specifying-the-files-to-distribute (which also states which files are included by default).


I understand that your current workflow has been optimised for years to work with the previous behaviour of setuptools. However the packaging ecosystem for Python has been changing for a while now. There are new standards in place and the setuptools wants to rely on these standards to allow us to remove complexity and be able to focus in other aspects of the implementation. This is one of the reasons why running setup.py has been deprecated.

If you want to keep the way things are current implemented, that is fine too! Both setuptools and pip have a lot of stuff implemented to support legacy packages. But you will have to be aware and workaround the limitations that are implied by that approach.

One of them is seeing the warning. You can still get around that by defining a filter via PYTHONWARNINGS.

On the other hand, if you want to give it a try to PEP 517, it might be worthy to start with the __legacy__ first to see how it goes. I also recommend looking how other projects (such as https://github.com/pybind/pybind11) deal with pre-built binaries and multiple setup_* files while still being able to rely on PEP 517.

You can always wrap an existing PEP 517 backends to tweak more deeply how they work for your package.

@mara004
Copy link

mara004 commented Feb 15, 2022

When I run:

rm -rf dist build src/*.egg-info
python setup.py sdist
tar tf dist/pypdfium2-0.12.0.tar.gz

I cannot find the platform_setup folder inside the distribution.

That is confusing. I cannot reproduce this behaviour, at least not with the latest state of the repository. When running your sequence of commands, I get a tarball including all directories and files that are not excluded in .gitignore (sample: pypdfium2-0.13.1.tar.gz).
However, as mentioned previously, setup.py is not intended for sdist, as it will copy platform-specific binary and bindings files into the src/pypdfium2/ directory. You need to use platform_setup/setup_source.py sdist instead.

I'll investigate the other mentioned points and reply in following comments.

@mara004
Copy link

mara004 commented Feb 15, 2022

If you want to keep the way things are current implemented, that is fine too!

It is not, because it means the code will eventually stop working, should the setuptools comman-line really be removed some day.

One of them is seeing the warning. You can still get around that by defining a filter via PYTHONWARNINGS.

The warning is there for a reason. Filtering it won't help anyone.

@mara004
Copy link

mara004 commented Feb 15, 2022

There are new standards in place and the setuptools wants to rely on these standards to allow us to remove complexity and be able to focus in other aspects of the implementation.

Sure, but the current approach of PEP 517 has serious limitations that cause a lot of problems for downstream packagers.
The rationale is very vague and theoretical; not sufficiently backed by practical use cases.

@abravalheri
Copy link
Contributor

One pratical use case is the future removal of easy_install.

@mara004
Copy link

mara004 commented Feb 15, 2022

I also recommend looking how other projects (such as https://github.com/pybind/pybind11) deal with pre-built binaries and multiple setup_* files while still being able to rely on PEP 517.

What pybind does looks interesting, but from what I can see it's not about dealing with pre-built binaries.
In addition to its main setup.py, pybind has setup_global.py.in and setup_main.py.in, but I don't quite understand under which circumstances these take effect and how they play together with the main setup.py.

@mara004
Copy link

mara004 commented Feb 16, 2022

There are new standards in place and the setuptools wants to rely on these standards to allow us to remove complexity and be able to focus in other aspects of the implementation. This is one of the reasons why running setup.py has been deprecated.

The problem is that this reduction in complexity of the setuptools codebase greatly increases the difficulty for downstream packagers. With the old setup.py approach, there was a lot of flexibility, and yet the interface could be kept relatively high-level so I only needed a small, easy to overview amount of code.
As PEP 517 essentially renders the high-level API unusable for cross-packaging, I will either have to resort to crude workarounds to be able to use the new tooling anyway, or use a low-level interface like Pep517HookCaller instead and write a lot of custom setup code.


Another general flaw of PEP 517 is that all projects need to add yet another setup file, pyproject.toml, in addition to the existing setup.py and setup.cfg. It might end up causing a lot more chaos and increased complexity than improvements. As Gentoo developer Michał Górny rightly metioned, pyproject.toml may also cause a bootstrap problem since the standard library does not provide a TOML parser:

«However, without a TOML parser in stdlib (and there’s no progress in providing one), this means that every single build system now depends on tomli, and involves a circular dependency. A few months back, every single build system depended on toml instead but that package became unmaintained. Does that make you feel confident?»

@abravalheri
Copy link
Contributor

abravalheri commented Feb 16, 2022

Hi @mara004, locally I was able to use python -m build to create both a .tar.gz and .whl files for pypdfium2, by implementing the changes I discussed in #2847 (comment) (specifically, adding platform_setup/__init__.py and a pyproject.toml with __legacy__ and setuptools-scm).

Of course this does not imply that those artifacts are correct (I am not an user of the project, nor I have knowledge about its code base and tests), in fact they can be completely wrong and just because I managed to get python -m build to work, it does not mean it did the right thing...

Maybe it might be interesting for you to give it a try and see if the tests run correctly after you install the artifacts built from python -m build?

Regarding PEP 517, I recommend bringing any discussions about its pitfalls and improvement proposals to https://discuss.python.org/c/packaging/14.

@mara004
Copy link

mara004 commented Feb 16, 2022

That's very interesting, thanks for trying this. Could you please submit a PR so I can take a look?

The test suite should be fairly easy to use: Just install the artifact that corresponds to your platform and run python3 -m pytest tests/. You can also inspect the artifacts with an archiver tool and see if the pypdfium2/ directory contains the files _pypdfium.py and one-of pdfium / pdfium.dll / pdfium.dylib, plus everything from src/pypdfium2/.

In particular, I would like to know how you managed to get python3 -m build use the platform-specific setup files?

@abravalheri
Copy link
Contributor

abravalheri commented Feb 16, 2022

In particular, I would like to know how you managed to get python3 -m build use the platform-specific setup files?

I don't know how to answer this question. What I really did was just add a platform_setup/__init__.py and the 3 line pyproject.toml file you can find in #2847 (comment). It does not even dignify a PR and most likely there are more changes needed to make it really work. But that might be a start for you guys to working with?

How does pip install . -v chooses the platform-specific setup files? I guess build just followed the same process...

@mara004
Copy link

mara004 commented Feb 16, 2022

Okay, I see. This means your changes do not solve the problem - I already tried your suggestions from #2847 (comment).
As outlined previously, I need to run the various setup files in platform_setup to create the 8 platform-specific wheels I upload to PyPI, but you created only a single wheel using setup.py, right?

How does pip install . -v chooses the platform-specific setup files?

Well, this is the procedure that setup.py goes through:

  • Ensures all dependencies are installed
  • Determines the host platform using sysconfig.get_platform()
  • Downloads the corresponding tarball from pdfium-binaries containing binaries and headers
  • Generates bindings for the headers using ctypesgen
  • Calls the shared setup function mkwheel_install() from setup_base with the platform data directory as argument.

However, the main point is that setup.py is only used for installation from git or for sourcebuild fallback. To PyPI, I upload platform wheels generated using the custom setup files, so setup.py is not running for end users who install a wheel.
(setup.py can't be used generally, for it relies on packaging dependencies that end users do not need at runtime, and I dind't even manage to get ctypesgen running on Windows.)

I guess build just followed the same process...

Yeah, but that misses the actual problem. Running the main setup.py using python3 -m build will work, that's out of question. The issue is that I need build to use the custom setup files to craft the release wheels in utilities/setup_all.sh. This is the part that I have to replace with PEP 517 tooling:

python3 platform_setup/setup_sdist.py sdist
python3 platform_setup/setup_darwin_x64.py    bdist_wheel
python3 platform_setup/setup_darwin_arm64.py  bdist_wheel
python3 platform_setup/setup_linux_x64.py     bdist_wheel
python3 platform_setup/setup_linux_arm64.py   bdist_wheel
python3 platform_setup/setup_linux_arm32.py   bdist_wheel
python3 platform_setup/setup_windows_x64.py   bdist_wheel
python3 platform_setup/setup_windows_x86.py   bdist_wheel
python3 platform_setup/setup_windows_arm64.py bdist_wheel

@mara004
Copy link

mara004 commented Feb 16, 2022

By the way, I now finally understood the sdist behaviour you reported: This is what happens if I uninstall setuptools-scm. I had it installed for other packages and never realised that sdist behaves differently if setuptools-scm is not available. I will thus list it in the packaging requirements.

@abravalheri
Copy link
Contributor

Yeah, but that misses the actual problem. Running the main setup.py using python3 -m build will work, that's out of question. The issue is that I need build to use the custom setup files to craft the release wheels in utilities/setup_all.sh:

This reminds me the way pybind11 uses an environment variable to select which setup file to use. Maybe you could try that?

@mara004
Copy link

mara004 commented Feb 16, 2022

This reminds me the way pybind11 uses an environment variable to select which setup file to use. Maybe you could try that?

How does that work? How can I make build use a different setup file? That's exactly my question.
It's not obvious to me what pybind's setup code does.

@abravalheri
Copy link
Contributor

Build will not do that automatically for you.
The guys in pybind11 came up with this strategy of using the main setup.py to select another file to run. It is quite ingenious.

They are exec-ing the secondary files, because they are creating them dynamically from templates, but in your case, since the files are static, you could simply use importlib.import_module and find the right function to run.

@mara004
Copy link

mara004 commented Feb 16, 2022

I see..... That's not a bad approach. It might be possible to do something similar for pypdfium2, though it would make the main setup file harder to overview.

@mara004
Copy link

mara004 commented Feb 18, 2022

@abravalheri I now changed pypdfium2 to use PEP 517 setup with an environment variable to control which code to run.
It may even be a bit better than before since dependencies can now be handled more nicely.
However, I currently still need "setuptools.build_meta:__legacy__" in pyproject.toml. I tried to remove the legacy requirement, but then pip3 install . does not work because importing from platform_setup fails, although I added an __init__.py file. However, it does work if using the option --no-build-isolation. It looks like the platform_setup directory may just not be copied into the isolated directory. I also tried to include it explicitly using a MANIFEST.in file, but couldn't get it to work either. Do you know how I could fix this, so the project does not depend on legacy behaviour?

@abravalheri
Copy link
Contributor

Hi @mara004, nice to hear that things seem to be working!

I think the problem with setuptools.build_meta (non legacy) is related to #2847 (comment): you cannot assume . (current working dir1) is added to sys.path (PYTHONPATH), so Python's import system doesn't know how to find platform_setup, even when/if it is copied to the isolated directory.

I imagine a quick and dirty way of doing that is sys.path.append(os.path.dirname(__file__)) before doing the imports from platform_setup. But people might find this too hacky2, and prefer using importlib.

Footnotes

  1. Being the CWD either the project directory or the directory used in isolated builds for copying the code.

  2. There might also be side-effects of manipulating sys.path directly, e.g. obfuscating packages used in build-time dependencies.

@mara004
Copy link

mara004 commented Feb 18, 2022

Ah, yes. Sorry, I overlooked you already mentioned this. I will see what I can do with importlib or sys.path.
You're right that just adding the CWD sys.path is not a nice solution, for it would include all the other files and subdirectories.

@mara004
Copy link

mara004 commented Feb 18, 2022

I think the following piece of code should work:

def include_platform_setup():
    
    name = 'platform_setup'
    PlatSetupInit = join(dirname(abspath(__file__)), name, '__init__.py')
    
    spec = importlib.util.spec_from_file_location(name, PlatSetupInit)
    module = importlib.util.module_from_spec(spec)
    
    sys.modules[name] = module
    spec.loader.exec_module(module)

with corresponding call of include_platform_setup() in main().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.
Projects
None yet
Development

No branches or pull requests

4 participants