Consider switching to TOML v1 compliant parser

[layday]

With several other projects (Poetry, by virtue of using tomlkit, ward) having switched or soon to switch (pip, Black) to a parser able to handle v1 of the spec, we might want to consider doing the same (or assisting with updating toml).

[FFY00]

This makes bootstrapping harder, as toml is already used by our dependencies and choosing a different parser would introduce yet another dependency to the tree. The PEP 518 spec does not specify TOML 1.0, so I believe it is fine to keep using this parser, though I acknowledge it would be better to use a TOML 1.0 compliant one.
We could provide a toml-1.0 extra until the toml library gains support for TOML 1.0 or until such parser is available in the standard library. This could be accompanied with a nice error message hinting that TOML parsing errors could be because pyptoject.toml is using a newer TOML version. How does that sound?

[layday]

This makes bootstrapping harder, as toml is already used by our dependencies and choosing a different parser would introduce yet another dependency to the tree.

Our only dependency which depends on toml is pep517. I think we could coordinate with @takluyver if this is a change we'd like to pursue.

The PEP 518 spec does not specify TOML 1.0, so I believe it is fine to keep using this parser, though I acknowledge it would be better to use a TOML 1.0 compliant one.

There's at least the inclination to update PEP 518 to require v1 of the spec.

We could provide a toml-1.0 extra until the toml library gains support for TOML 1.0 or until such parser is available in the standard library. This could be accompanied with a nice error message hinting that TOML parsing errors could be because pyptoject.toml is using a newer TOML version. How does that sound?

I think we can probably wait this one out. Once pip adops tomli, other projects will follow suit. Until then there's no rush I would think.

[takluyver]

I'm assuming it will change at some point, but I don't personally feel much pressure to change it soon. I appreciate that projects using Poetry might be keener.

I would really like to see a basic TOML parser in the standard library (tomli looked to me like a good candidate for this), to avoid bootstrapping issues. Currently flit_core defines its own metadata in Python code so it can be bootstrapped without a TOML parser.

[gaborbernat]

I would really like to see a basic TOML parser in the standard library

Per the latest discussion, this feels to me unlikely to happen - https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/64. Also, even if it does happen, that's at least three years away from being broadly available. I don't think you want to wait that long to solve this issue.

[takluyver]

I appreciate that adding things to the standard library is a very long-term solution, and I'm OK with that. And there are probably options that we wouldn't accept as permanent solutions but might be OK with for the interim until we can rely on a parser in the stdlib.

[gaborbernat]

What if that day never comes? Per my link above to me seems very likely there's not enough agreement to add a parser into the stdlib.

[takluyver]

What can I say? Obviously I'll live with it if core dev never want to add anything to the standard library again. I'm just expressing that I hope they do, and a TOML parser seems like a really good thing to add. I also haven't got the impression that it's as foregone a conclusion as you suggest that it won't be accepted.

[gaborbernat]

I wasn't saying it won't be added, but it's 50%-50% for now. As such we should plan as much for the possibility they add as they don't.

[layday]

Not sure what the relevance of all of this is - does a TOML parser not being part of the stdlib prevent us from using another, v1-compliant parser? I'm personally not too enthusiastic about leaving a library which has served us well for years behind, but if the community is intent on moving in that direction, the alternative is to say our toolset is incompatible with pip and others.

[takluyver]

Sorry, yup, got sidetracked a bit. 😐

If pip switches to tomli, I'll likely follow that for my projects - I like the focus on a small API for parsing only.

The wrinkle for pep517 is that it still supports old Python versions, including 2.7, as a foundational piece of infrastructure. But all the pieces it actually uses a toml library for are deprecated in favour of the caller loading the toml and passing values in. So maybe it doesn't matter.

[layday]

Sorry, yup, got sidetracked a bit. 😐

No worries, was thinking I might've missed something.

[layday]

pip, black, setuptools_scm, pytest, towncrier and coverage have all switched to tomli now.

[FFY00]

The packages I care about for the consideration in this project are setuptools and pep517.

[layday]

build is often used in conjunction with the projects I listed above. If build can't read a pyproject.toml which all of these other tools can read that is obviously relevant to us.

[FFY00]

We can softdepend on tomli as a preference over toml, but toml should be the fallback as it is the one used by the projects I listed and choosing something else would make bootstrapping harder.

[layday]

Well, @takluyver has said that he'd follow the lead of pip (#308 (comment)). setuptools does not use TOML.

[henryiii]

setuptools does not use TOML.

What about when adding PEP 621 support?

[gaborbernat]

What about when adding PEP 621 support?

What has this project to do with PEP-621? Isn't that a backend thing?

[FFY00]

I think Henry meant that setuptools will adopt a parser soon for PEP 621 support.

[layday]

I assume when that time comes - should it ever come - setuptools will not choose a TOML parser which is abandoned. setuptools is not a runtime dependency of build regardless. I suggest switching to tomli after both pypa/pyproject-hooks#118 and #324 have been merged - no soft dependency on tomli required.

[takluyver]

I'm kind of amazed how quickly we've gone from 'there's this project called tomli' to 'that's the standard Python toml parser now'. 🙂 Hopefully now that there's a TOML v1 spec and a parser with a deliberately minimal API, the situation will stabilise a bit.

Anyway, as I said before, I do mean to follow this change in projects I maintain, including pep517.