Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

abravalheri · 2021-12-25T19:20:44Z

Summary of changes

Based on the experience acquired in #2924, I created this PR as a simplification (please check the discussion there for details).
There are many individual steps, but the following image illustrates how they are organised in 4 main groups of stacked changes:

a) abravalheri/setuptools@1a1397a...4a98608: Update build_meta to use dist objects instead of simply running setup.py
b) abravalheri/setuptools@4a98608...7fe7a41: Separate setup.cfg parsing and extract common post-processing functions

(new vendorised packages)

c) abravalheri/setuptools@fab3b79...5666556: Add means to apply configuration from pyproject.toml files
d) abravalheri/setuptools@7fe7a41...d34794d: Integrate pyproject.toml configuration into existing classes

Finally there is a completely non-required group of changes that only adds some type hints to improve (my) level of confidence in the implementation of the setuptools.config sub-package:
e) abravalheri/setuptools@d34794d...0aff4ea: Add some type hints to the setuptools.config subpackage

Unfortunately GitHub does not provide tools to facilitate the review of stacked changes/PRs, but I am open to suggestions on how to facilitate the review. I hope that this organisation in groups makes sense and help in the process.

I also have another branch with a proof-of-concept where I remove setuptools' own setup.cfg and add all the existing configurations in the pyproject.toml.

Closes #1688, #2671 (mostly... see discussion bellow).

This PR adds 2 vendorised dependencies: tomli and validate-pyproject.

Pull Request Checklist

Changes have tests
News fragment added in changelog.d/.
(See documentation for details)

abravalheri · 2021-12-25T19:21:36Z

Differences from #2924

In #1688 the proposed approach (which have some level of consensus) is to deprecate configuration via setup.cfg files but keep supporting them by automatically converting setup.cfg into pyproject.toml and then proceed with the parsing.

In PR #2924, I try to not only completely follow this approach but also added a series of safe-guards to catch regression problems and ensure backward compatibility. This made the PR very complex and not review-friendly...

My proposal now is to avoid performing this automatic conversion, because it reduces complexity. The internals of setuptools are not prepared to deal with metadata and configs in the way PEP 621 specifies, so the automatic conversion seems a bit pointless, as if we go to the from the INI-state-space to TOML-state-space and then back to something very similar to the INI-state-space.

I think the best would be first re-working the internals of setuptools to use a more modern version of the Core Metadata spec, and then proceed with the automatic conversion.

Another difference is that in #2924 I try to use the latest version of the Core Metadata spec as "lingua franca" between pyproject.toml and the existing way of configuring setuptools. So it should also be possible to scavenge the code in #2924 to help updating support for Core Metadata (but that is left as a future work).

Please note I am providing an external package (abravalheri/ini2toml) anyway to assist in the migration. This package can still be used in the future.

abravalheri · 2021-12-25T19:22:01Z

Setuptools-specific `tool` table in `pyproject.toml`

As many have previously pointed out PEP 621 does not specify many of the configuration fields required by setuptools to work properly, which seems to indicate that inevitably the users would have to use a non-standardised [tool.setuptools] sub-table in pyproject.toml (and that would "undermine" the efforts for migrating the configuration format).

I would argue that, this is not necessarily the case. As per the implementation in this PR, any configuration coming from pyproject.toml will consider include_package_data=True when the field is absent, as opposed to the behaviour in setup.cfg (which is preserved for backward compatibility).

If we sum to this the changes proposed in #2894 for automatically auto-discovering flat and src-layouts, I believe that most of the packages will be able to use pyproject.toml without a [tool.setuptools] sub-table. (Users will still be able to use it for further customisations or to declare dynamic values to be backfilled by setuptools).

The equivalence between setup.cfg and pyproject.toml and the intuition to convert between them is explained in the ini2toml documentation.

abravalheri · 2021-12-25T19:22:10Z

Proof-of-concept converting setuptools' own `setup.cfg` file to `pyproject.toml` and blockers

As I previously mentioned, in this branch a version of setuptools that does not have a setup.cfg (but instead uses pyproject.toml for its own configuration) can be found.

I decided to not include it in this PR because of the following blockers:

I believe that the scripts in the tools/ folder that automate releases currently can only handle setup.cfg (e.g. when bumping the version).
Using a top-level table for pytest in pyproject.toml offends PEP 518 jaraco/pytest-enabler#4 (which will cause errors when setuptools runs the validations based on validate-pyproject

abravalheri · 2021-12-25T19:29:10Z

Future Works

The following tasks/themes can be further explored in future PRs (either by me or other member of the community):

Include documentation about how to use pyproject.toml to configure the distribution
Update the version of Core Metadata produced by setuptools.

(These were excluded from this PR to avoid increasing even more its size).

abravalheri · 2022-01-04T22:35:31Z

setuptools/config/expand.py

+
+
+def find_packages(
+    *, namespaces=False, root_dir: Optional[_Path] = None, **kwargs


Should namespaces default to True here? (It makes sense since implicit namespaces are standardised with a PEP...)

We can implement it in a backwards compatible way by tweaking setuptools.config.setupcfg, this way configuration coming from setup.cfg files or setup.py will still default to namespaces=False, but new configuration coming from pyproject.toml will default to namespaces=True...

I have been wanting to transition to supporting namespace packages by default, so +1 to the proposal.

It was suggested by @webknjaz that we address this transition in a follow up comment.

abravalheri · 2022-01-04T22:41:36Z

setuptools/config/pyprojecttoml.py

+        value = _expand_dynamic(dynamic_cfg, field, package_dir, root_dir, silent)
+        project_cfg[field] = value
+
+    if "version" in dynamic and "version" in dynamic_cfg:


This value can be given by extensions such as setuptools-scm, so we don't have to fail if no configuration for version is given in [tool.setuptools.dynamic].

setuptools/tests/test_build_meta.py

setuptools/tests/test_editable_install.py

abravalheri · 2022-01-10T09:23:02Z

setuptools/config/expand.py

@@ -0,0 +1,330 @@
+"""Utility functions to expand configuration directives or special values


This file was originally extracted from the existing implementation of setuptools.config and after that some improvements were added (e.g. allowing explicitly specifying a project root directory)

marvinpfoertner · 2022-01-24T14:47:26Z

@abravalheri First of all, thank you for taking care of this, this is a feature I've been wanting for a while now :) If you don't mind me asking: what is the status of this PR? I plan on integrating this into my projects as soon as possible.

abravalheri · 2022-01-24T14:50:52Z

Hi @marvinpfoertner thanks for the kind words. Since this is a complex change in setuptools, I am waiting for when other maintainers might have some time to review it.

If you want to help, you are more than welcome to leave your review also 😄

I will rebase the code later this week with the latest version of main to get rid of the conflict files.

marvinpfoertner · 2022-01-24T15:07:17Z

Hi @marvinpfoertner thanks for the kind words. Since this is a complex change in setuptools, I am waiting for when other maintainers might have some time to review it.

If you want to help, you are more than welcome to leave your review also smile

I will rebase the code later this week with the latest version of main to get rid of the conflict files.

Thanks for the quick reply! :) I'm not sure if I can contribute anything with my review, since I have absolutely no idea about the internals of setuptools 😀

pradyunsg · 2022-01-24T23:29:06Z

changelog.d/2970.change.1.rst

+Please note that the legacy backend is not guaranteed to work with
+``pyproject.toml`` configuration.


Can we instead make it a clean unambigous failure? One of the frustating properties of working with distutils/setuptools is that there are a lot of ways to get subtle failures / accidental behaviours, and it'd be nice to eliminate a potential class of issues here by simply making it an explicit failure.

We can aways relax things later, if we so choose, so it'd be nice to start with strict and clear boundaries.

Thank you very much for the review @pradyunsg.

I have nothing against raising a proper exception here. Just please note that right now there are no intrinsic problems with using pyproject.toml configurations with the legacy backend, I just wanted to make explicit that this is not part of the supported interface.

That aside, do you have any recommendation on how to implement this? Right now the configuration files are being read inside setuptools.dist:Distribution.parse_config_files, which is unaware about which backend was chosen.

For the "non-legacy" backend, in this PR I managed to avoid the "blind execution" of the setup.py script, so we can have more control about how/when all these functions are called. However for the "legacy" backend I purposefully wanted to keep the execution as backwards compatible as possible, so all parsing related functions are invoked via distutils.core, which is not easily changeable...

bootstrap.egg-info/entry_points.txt

jaraco

Overall, this looks good to me. I haven't had a chance to review it in depth, and I don't know when I will, but I'd like to get it out so users can experiment with it, but also to identify any regressions in the existing behaviors.

How difficult would it be to present this feature as opt-in, so the new behavior isn't available by default but is only available with a feature flag, like SETUPTOOLS_ENABLE_FEATURES="PEP 621". That way, users can experiment with it but would recognize that the feature isn't universally available.

KOLANICH · 2022-01-30T21:39:47Z

I'm against of putting it under an env variable. If it is not ready, it shouldn't land into main. If it is ready, it should be just available for everyone who has installed the needed version of setuptools. Installing any version having this feature, be it installing a version from this branch, or installing one from main or installing any release supporting this feature, is by itself the opt-in.

blink1073 · 2022-01-30T21:53:18Z

If it were behind an env variable, would that not affect a consumer of an sdist unless they also knew to set that variable?

jaraco · 2022-01-31T03:30:10Z

If it were behind an env variable, would that not affect a consumer of an sdist unless they also knew to set that variable?

Yes that's right. It would only be available for local testing/experimentation. I'd like to validate the use-case against some packages before users come to depend on it.

I'm against of putting it under an env variable. If it is not ready, it shouldn't land into main. If it is ready, it should be just available for everyone who has installed the needed version of setuptools. Installing any version having this feature, be it installing a version from this branch, or installing one from main or installing any release supporting this feature, is by itself the opt-in.

If we go with that approach, users could upload packages that depend on the feature. Users who happen to use the feature before it's stable might find those packages no longer install if a later version removes the feature or makes a breaking change to it. I just want to emphasize the message that this feature is experimental and projects shouldn't depend on it until it's proven stable.

KOLANICH · 2022-01-31T08:09:26Z

If we go with that approach, users could upload packages that depend on the feature.

The same way they can upload packages depending on libs that are non-public.

Users who happen to use the feature before it's stable might find those packages no longer install if a later version removes the feature or makes a breaking change to it.

it is OK, the features not yet landed into a release are explicitly unstable.
Even if they had landed, it would be OK to break compatibility if it is really needed.

I just want to emphasize the message that this feature is experimental and projects shouldn't depend on it until it's proven stable.

We can emit a warning to make everyone aware the feature is unstable.

I personally am going to convert all my projects to PEP 621 as soon as the feature appears in main (if editable install is working). In an unlikely scenario of anything changing I will just write a script fixing all my projects. I have already done things like that.

agronholm · 2022-03-06T23:00:16Z

Ok, I tried this again with the latest changes, and the build passed. I got two warnings though:

warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

I have a bunch of project URLs defined – is "url" really required?
Also, I have this: authors = [{name = "Alex Grönholm", email = "alex.gronholm@nextday.fi"}], so what is it complaining about?

moshez · 2022-03-06T23:04:00Z

@agronholm -- I got the same warnings.

Note that the package is created correctly: https://github.com/moshez/pyproject-only-example/tree/a79f89201915a4c2ac1cf2c14f370cc29449884d resulted in https://pypi.org/project/orbipatch/2022.3.6.1/ (note that the Homepage and author links are correct). The warnings are spurious.

agronholm · 2022-03-06T23:05:44Z

Yup, I was just looking at the generated PKG-INFO when you posted that. Indeed it seems the metadata is generated correctly.

abravalheri · 2022-03-07T00:13:52Z

Hi @moshez and @agronholm , thank you very much for testing it out!

The warning with the author is related to this: pypa/distutils#116

I still need to have a look on the URL one 😅

moshez · 2022-03-07T00:20:52Z

@abravalheri -- thank you for the explanation, link, and context. In order to avoid multiple people reporting the same issues, do you think it would make sense to edit your top-level post in https://discuss.python.org/t/help-testing-experimental-features-in-setuptools/13821 with "Known issues:"?

abravalheri · 2022-03-07T10:31:02Z

Hi @agronholm, I am investigating the url warning.

Could you tell me what are the URLs you have defined? I think this warning also comes directly from distutils if you don't have a "homepage". (It can be "Home-page", "homepage", "home-page" etc...)

abravalheri · 2022-03-07T10:51:22Z

do you think it would make sense to edit your top-level post in https://discuss.python.org/t/help-testing-experimental-features-in-setuptools/13821 with "Known issues:"?

Thanks @moshez, I just updated the post: https://discuss.python.org/t/help-testing-experimental-features-in-setuptools/13821#currently-known-issuesnon-intuitive-behaviour-not-blockers-4

abravalheri · 2022-03-22T16:13:16Z

To those following this PR:

Thank you very much for all the feedback!
As some might have noticed, I have created a series of follow-up PRs that include some initial docs, better adherence to PEP 621 in terms of the handling of the license field, auto-discovery, etc... These PRs were merged to experimental/support-pyproject.

My plan is to merge experimental/support-pyproject into main soon (which should automatically close this PR), but preserve the experimental status for the new features, so we can still react to user feedback and examine more thoroughly the design of the [tools.setuptools] table.

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. This change also enables support for Wagtail 3.x by removing the `<3` version pin. This will result in the last release of TreeModelAdmin to support Wagtail < 4. Wagtail 4.x is not yet supported. The version is bumped to 1.5.0 for anticipated release to permit installing with Wagtail 3.x.

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. Because [flake8 does not want to support pyproject.tml yet](PyCQA/flake8#234), unlike all the other tools we use, I’m removing it in favor of [ruff](https://github.com/charliermarsh/ruff). This change also enables support for Wagtail 3.x by removing the `<3` version pin. This will result in the last release of TreeModelAdmin to support Wagtail < 4. Wagtail 4.x is not yet supported. The version is bumped to 1.5.0 for anticipated release to permit installing with Wagtail 3.x. I’ve also brought in some quality-of-life improvements for coverage checking from the Django-Flags tox/GHA setup.

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. Because [flake8 does not want to support pyproject.tml yet](PyCQA/flake8#234), unlike all the other tools we use, I’m removing it in favor of [ruff](https://github.com/charliermarsh/ruff).

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. Because [flake8 does not want to support pyproject.tml yet](PyCQA/flake8#234), unlike all the other tools we use, I’m removing it in favor of [ruff](https://github.com/charliermarsh/ruff). Co-authored-by: Andy Chosak <andy.chosak@cfpb.gov>

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. Because [flake8 does not want to support pyproject.tml yet](PyCQA/flake8#234), unlike all the other tools we use, I’m removing it in favor of [ruff](https://github.com/charliermarsh/ruff). This change also continues support for Wagtail 3.

With [PEP 621 support added to setuptools](pypa/setuptools#2970) last year, and with pip supporting editable installs as of 21.3, we can move to `pyproject.toml` and deprecate `setup.py`. The file remains for legacy pip support. Because [flake8 does not want to support pyproject.tml yet](PyCQA/flake8#234), unlike all the other tools we use, I’m removing it in favor of [ruff](https://github.com/charliermarsh/ruff). This change also continues support for Wagtail 3. Co-authored-by: Andy Chosak <andy.chosak@cfpb.gov>

This comment has been minimized.

Sign in to view

abravalheri marked this pull request as ready for review December 25, 2021 20:33

abravalheri mentioned this pull request Jan 4, 2022

Add support for pyproject.toml-style configuration (PEP 621) #2924

Closed

2 tasks

abravalheri commented Jan 4, 2022

View reviewed changes

setuptools/tests/test_build_meta.py Outdated Show resolved Hide resolved

abravalheri commented Jan 4, 2022

View reviewed changes

setuptools/tests/test_build_meta.py Outdated Show resolved Hide resolved

abravalheri commented Jan 4, 2022

View reviewed changes

setuptools/tests/test_editable_install.py Outdated Show resolved Hide resolved

abravalheri mentioned this pull request Jan 5, 2022

[BUG] ModuleNotFoundError when using cmdclass option in PEP517-style build #3000

Closed

1 task

abravalheri force-pushed the support-pyproject branch from 281e51d to f97d27a Compare January 7, 2022 00:36

abravalheri mentioned this pull request Jan 10, 2022

Discussion: support for pyproject.toml configuration #1688

Closed

abravalheri commented Jan 10, 2022

View reviewed changes

JulienPalard mentioned this pull request Jan 12, 2022

Implement PEP 621 in setuptools -- Storing project metadata in pyproject.toml psf/fundable-packaging-improvements#25

Closed

hukkin mentioned this pull request Jan 17, 2022

Compile code with Mypyc hukkin/tomli-w#33

Open

abravalheri force-pushed the support-pyproject branch from b853892 to 38758f5 Compare January 24, 2022 17:34

pradyunsg reviewed Jan 24, 2022

View reviewed changes

Kludex mentioned this pull request Jan 29, 2022

🛠 Refactored setup.py to setup.cfg encode/starlette#1359

Closed

jaraco reviewed Jan 30, 2022

View reviewed changes

bootstrap.egg-info/entry_points.txt Outdated Show resolved Hide resolved

jaraco approved these changes Jan 30, 2022

View reviewed changes

nullableVoidPtr mentioned this pull request Mar 9, 2022

Migrate to PEP 518 qilingframework/qiling#1111

Closed

15 tasks

abravalheri added this to the Add experimental support for project metadata via `pyproject.toml` milestone Mar 17, 2022

erwanp mentioned this pull request Mar 20, 2022

fixes invalidindex error radis/radis#428

Merged

abravalheri merged commit 7ff3dc1 into pypa:main Mar 24, 2022

di mentioned this pull request Mar 24, 2022

Move to metadata to pyproject.toml pypa/sampleproject#166

Merged

willbarton mentioned this pull request Dec 20, 2022

Support Wagtail 3.x and switch to pyproject.toml cfpb/wagtail-treemodeladmin#34

Merged

5 tasks

willbarton mentioned this pull request Jan 11, 2023

Switch to pyproject.toml and add a tox -e interactive environment cfpb/wagtail-flags#71

Merged

Shixiaowei02 mentioned this pull request Apr 11, 2024

pip install -e . does not work NVIDIA/TensorRT-LLM#1065

Closed

4 tasks

ue-sho mentioned this pull request May 13, 2024

Change to use pipenv KitwareMedical/dicom-anonymizer#81

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021

This comment has been minimized.

abravalheri Jan 4, 2022

jaraco Jan 30, 2022

abravalheri Feb 11, 2022

abravalheri Jan 4, 2022

abravalheri Jan 10, 2022

marvinpfoertner commented Jan 24, 2022

abravalheri commented Jan 24, 2022 •

edited

marvinpfoertner commented Jan 24, 2022

pradyunsg Jan 24, 2022 •

edited

abravalheri Jan 25, 2022

jaraco left a comment

KOLANICH commented Jan 30, 2022 •

edited

blink1073 commented Jan 30, 2022 •

edited

jaraco commented Jan 31, 2022

KOLANICH commented Jan 31, 2022 •

edited

agronholm commented Mar 6, 2022

moshez commented Mar 6, 2022

agronholm commented Mar 6, 2022

abravalheri commented Mar 7, 2022 •

edited

moshez commented Mar 7, 2022

abravalheri commented Mar 7, 2022

abravalheri commented Mar 7, 2022

abravalheri commented Mar 22, 2022



		def find_packages(
		, namespaces=False, root_dir: Optional[_Path] = None, *kwargs

		@@ -0,0 +1,330 @@
		"""Utility functions to expand configuration directives or special values

		Please note that the legacy backend is not guaranteed to work with
		``pyproject.toml`` configuration.

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

Conversation

abravalheri commented Dec 25, 2021 • edited

Summary of changes

Pull Request Checklist

abravalheri commented Dec 25, 2021 • edited

Differences from #2924

abravalheri commented Dec 25, 2021 • edited

Setuptools-specific tool table in pyproject.toml

abravalheri commented Dec 25, 2021 • edited

Proof-of-concept converting setuptools' own setup.cfg file to pyproject.toml and blockers

abravalheri commented Dec 25, 2021

Future Works

This comment has been minimized.

abravalheri Jan 4, 2022

Choose a reason for hiding this comment

jaraco Jan 30, 2022

Choose a reason for hiding this comment

abravalheri Feb 11, 2022

Choose a reason for hiding this comment

abravalheri Jan 4, 2022

Choose a reason for hiding this comment

abravalheri Jan 10, 2022

Choose a reason for hiding this comment

marvinpfoertner commented Jan 24, 2022

abravalheri commented Jan 24, 2022 • edited

marvinpfoertner commented Jan 24, 2022

pradyunsg Jan 24, 2022 • edited

Choose a reason for hiding this comment

abravalheri Jan 25, 2022

Choose a reason for hiding this comment

jaraco left a comment

Choose a reason for hiding this comment

KOLANICH commented Jan 30, 2022 • edited

blink1073 commented Jan 30, 2022 • edited

jaraco commented Jan 31, 2022

KOLANICH commented Jan 31, 2022 • edited

agronholm commented Mar 6, 2022

moshez commented Mar 6, 2022

agronholm commented Mar 6, 2022

abravalheri commented Mar 7, 2022 • edited

moshez commented Mar 7, 2022

abravalheri commented Mar 7, 2022

abravalheri commented Mar 7, 2022

abravalheri commented Mar 22, 2022

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021 •

edited

abravalheri commented Dec 25, 2021 •

edited

Setuptools-specific `tool` table in `pyproject.toml`

abravalheri commented Dec 25, 2021 •

edited

Proof-of-concept converting setuptools' own `setup.cfg` file to `pyproject.toml` and blockers

abravalheri commented Jan 24, 2022 •

edited

pradyunsg Jan 24, 2022 •

edited

KOLANICH commented Jan 30, 2022 •

edited

blink1073 commented Jan 30, 2022 •

edited

KOLANICH commented Jan 31, 2022 •

edited

abravalheri commented Mar 7, 2022 •

edited