Discussion: support for pyproject.toml configuration

[pganssle]

Over the last few months, I've been using setup.cfg more, and I find the syntax and types and such to be super counter-intuitive. I don't find TOML to be a breeze per se, but it's at least a lot clearer to me how to specify a string, a list, etc. We also hear a lot of cries to move towards supporting pyproject.toml as the "one file" for setup configuration, in favor of setup.cfg. I'm not terribly opposed to this, but I want to open up the discussion for a few options:

1. Add pyproject.toml as the one true way to do declarative builds, moving all setuptools configuration over there and deprecating setup.cfg (partially what is discussed in Merge setup.cfg spec with pyproject.toml one and deprecate setup.py and setup.cfg #1160, and if we decide on that option we can move over to that issue).
2. Add pyproject.toml as a second supported way to do a declarative configuration, alongside setup.cfg.
3. Stay the course with setup.cfg indefinitely.

I am deeply concerned with the large amount of churn that we're introducing into the Python packaging space, so I don't want to make a lot of changes for the sake of changes, but I do think that at least having the option to use a pyproject.toml would be a real benefit, so I'm basically in favor of option 2.

I think the biggest downside of option 2 is that it means maintaining, testing and documenting three separate locations for all of our options - how to do it in setup.cfg, how to do it in pyproject.toml, and how to do it in setup.py. I think the hard part is going from 1 way to do everything to 2 ways. Our documentation is a mess anyway, but my beautiful vision for the future is that all our examples have "tabs" that allow you to toggle them between setup.cfg and setup.py. Adding a pyproject.toml in there would probably not be the worst thing.

[jaraco]

I'd prefer option 1. I imagine during the deprecation period that the setup.cfg documentation could be removed, but a permalink to the old documentation made available for projects still relying on that technique.

One way to ease the transition could be for the setup.cfg code to generate a .toml file and consume then have the pyproject.toml consumer just load both files. That way, a project seeking to transition would only need to do one build, manually merge the two .toml files, and delete setup.cfg.

My second choice would be option 3. I'd really like to avoid supporting multiple formats that do essentially the same thing.

Another consideration I'd like to mention is the proposal I only recently started fleshing out in pypa/packaging-problems#248. Such an approach would be yet another declarative format. However, this proposed approach would enable project developers to supply metadata decoratively in the canonical format (or in a format suitable for the requisite build steps). If such an approach is deemed viable and comes to fruition, the value in migrating to pyproject.toml may prove small. I suggest postponing this transition at least until that more ambitious proposal is disposed or reaches fruition.

[njsmith]

I imagine during the deprecation period that the setup.cfg documentation could be removed, but a permalink to the old documentation made available for projects still relying on that technique.

I'm pretty sure the deprecation period would be literally forever. I bet there are projects on PyPI that (a) are dead and will never see another release, (b) are still used, and (c) depend on setup.cfg to install.

I'm not opposed to adding features to setuptools and better ways to do things, but I think it's about 100x less important than not breaking stuff.

[pradyunsg]

I'm in favour of 1, with deprecated forever status (ideally keep the heading in documentation and just remove all the content, replacing it with a paragraph saying use pyproject.toml)

[qlixed]

There is an ETA for this? Or there is still some discussion pending?.I think that Option 1 is the best, but here is not stated for sure if this will be done.

[pganssle]

@qlixed There is no ETA.

I think we probably have a rough consensus for Option 1 with an indefinite deprecation period (we don't have to specify the length of the deprecation period anyway). I won't have time to do this any time soon, but I think we can accept a contribution that does.

I think we'll need to:

1. Vendor the toml library
2. Build out a tool that converts setup.cfg to pyproject.toml
3. Make it the default way that setup.cfg is processed.

I think we can say that if options are specified in both setup.cfg and pyproject.toml (under tool:setuptools), that we throw an error and refuse to build (use one or the other, not both). I don't think we should warn if people use setup.cfg even when pyproject.toml is available, we should wait for pyproject.toml configuration to be stable for at least a year before actively pushing it like that.

We'll also need to decide how to handle the little ad-hoc "types" we've implemented like file:, attr:, etc. Are we parsing these as strings (I really hate how much string parsing we do in setup.cfg as it is)? Using a dict? Is there some facility in the TOML language that would make this easy for us?

[layday]

Is there some facility in the TOML language that would make this easy for us?

Yes, TOML supports both nested and inline tables. For example, the version could be hardcoded as a string:

version = "0.0.0"

... or it could be a table specifying one of attr or file:

version = { attr = "setuptools.version" }

The same 'trick' is used by Poetry to separate dependency markers from version specifiers.

[brainwane]

In the fundable packaging improvements list I summarized this as follows (if it's inaccurate, I'd appreciate a correction, especially as a pull request):

Add support for pyproject.toml as a way to configure setuptools

setuptools does not yet allow project creators to use the new pyproject.toml standard configuration file to configure setuptools behavior. This distracts and confuses package creators, and prevents platforms and tools from depending on the presence of standard pyproject.toml metadata in packages. We'd like to implement pyproject.toml configuration support in setuptools. This requires backend development work, technical writing, and coordination and publicity work among Python users.

I'd also like to better work out what this entails and how long it would take. To dig into @pganssle's proposed sequence:

I think we'll need to:

1. Vendor the `toml` library

2. Build out a tool that converts `setup.cfg` to `pyproject.toml`

Would this live within setuptools or be part of some other codebase?

3. Make it the default way that `setup.cfg` is processed.

One subtask being: decide and implement how to deal with it if people use setup.cfg even when pyproject.toml is available.

Also: should this task include implementing a deprecation notices to output when the user's supplied a setup.cfg file, advising them to switch to pyproject.toml in the future?

So that's the backend development work. I'd add in:

4. technical writing, and coordination and publicity work among Python users:

• update all the bits of PyPUG that mention setup.cfg
• announce on Discourse & distutils-sig and deal with any questions/concerns that come up from other packaging tools maintainers
• write/update user help for https://setuptools.readthedocs.io/
• get user testing for the change (for instance, look around GitHub for issues that stem from setuptools not having this feature, and point them at the branch to test it)
• once it's landed in a released version, advertise a bit with some blog posts, Twitter threads, conference talks, and podcast appearances

What have I missed or gotten wrong? Is this mostly right?