Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twine check should guard against python_requires conflicting with the wheel tags #739

Open
webknjaz opened this issue Mar 8, 2021 · 45 comments
Labels
blocked Issues we can't or shouldn't get to yet feature request question Discussion/decision needed from maintainers

Comments

@webknjaz
Copy link
Member

webknjaz commented Mar 8, 2021

The Issue

With the sunset of Python 2, distribution maintainers should stop producing universal wheels with (ones with py2.py3 tag).

The solution is to drop universal = 1 or turn that into 0 (pypa/packaging.python.org#726) but it's easy to miss this bit. So this results in people adding things like python_requires = >= 3.6 but still having that py2 tag in the wheels which is rather useless (and, well, incorrect).

The same could happen to platform-specific wheels specifying ABI3 with tags like abi3-py36 while having python_requires = >= 3.7, for example.

I propose improving the twine check command to detect such inconsistencies.

@asottile
Copy link
Contributor

I'd like to voice my -1 -- universal just means "pure python and works on any version" and presumably will hold the same meaning when C-incompatible python 4.x rolls around. For the same reason that python_requires = <4 is bad I don't think this should be implemented.

@webknjaz
Copy link
Member Author

@asottile that's what I was thinking until @gaborbernat pointed out that it's not exactly true. It is currently py2.py3 in the wheel names. How do you suggest we should lint the conflicting metadata?

@asottile
Copy link
Contributor

It is true, or at at least what the documented point of --universal was: pypa/wheel@4a1b292#diff-8d068e8797e88947c320f79e856c3e16a72b730124a8f9d7031e2c4680dfa534R110-R111

the metadata isn't conflicting, pip properly handles python_requires and will only consider the wheel on appropriate python 3 versions

@webknjaz
Copy link
Member Author

equates to the tag "py2.py3".

This is hinting pip to attempt downloading the wheel under Python 2. @pradyunsg seems to think that this could cause problems with the old resolver (combined with Requires-Python: >=3.6, for example) because it doesn't know how to do backtracking.

@webknjaz
Copy link
Member Author

Also, I think that it's not the config for the tool producing wheels that should be linted but the resulting metadata in the wheel.

@gaborbernat
Copy link

I'd like to voice my -1 -- universal just means "pure python and works on any version" and presumably will hold the same meaning when C-incompatible python 4.x rolls around. For the same reason that python_requires = <4 is bad I don't think this should be implemented.

To achieve that intent the wheel tag this should map would need to be just py, not py2.py3 . because py2.py3 is not python 4 compatible tag.

@asottile
Copy link
Contributor

It would only cause problems for pip versions that don't understand python_requires (<9) -- which at this point they have much much harder problems installing valid packages

what I'd rather see is if python2 is truly dead then wheel / pip / setuptools should stop producing py2 in the name instead of requiring me to remove valid metadata from my configuration (which I use to signal that my package is pure python) I'll have to annoyingly add back later

@asottile
Copy link
Contributor

To achieve that intent the wheel tag this should map would need to be just py, not py2.py3 . because py2.py3 is not python 4 compatible tag.

I like this idea! let's push to make that happen

@gaborbernat
Copy link

gaborbernat commented Mar 10, 2021

I mean, either universal should start meaning py or python 3+ packages should stop using it. The current universal translation to py2.py3 means generated packages end up in an undesirable state because they are incompatible with python 2. At the same time, they contain code that works only with Python 3+.

Can we get some opinion from at least:

  • @pfmoore - as BDFL for packaging
  • @pradyunsg - as pip maintainer (where this tag is consumed)
  • @agronholm - as wheel maintainer (where this setting is used)

Thanks!

@agronholm
Copy link

I think pypa/wheel#396 is on the right track, except that I would rather abort the operation if such a discrepancy is detected because otherwise this could go unnoticed by the users.

@pfmoore
Copy link
Member

pfmoore commented Mar 10, 2021

As packaging BDFL, the wheel spec doesn't mention "universal" at all, so I'd consider that an implementation detail of the wheel package.

From pip's POV, we consume the tags, not "universal" directly. I'd be happy if pip were to add a check that the tags and Python-Requires were consistent, but I'd see that as a quality of implementation matter. For correctness, what matters is that pip only considers wheels with supported tags, and it rejects any where Python-Requires isn't satisfied. That's well-defined behaviour even if the tags and python-requires conflict.

@agronholm
Copy link

Yep, universal is an implementation detail in the wheel package. This feature will be removed in the eventual 1.0.0 release. In the meantime, raising an error when a conflict is detected between python_requires and universal is my current game plan.

@gaborbernat
Copy link

Yep, universal is an implementation detail in the wheel package. This feature will be removed in the eventual 1.0.0 release. In the meantime, raising an error when a conflict is detected between python_requires and universal is my current game plan.

This will break a lot of CIs, rightfully though.

@agronholm
Copy link

This will break a lot of CIs, rightfully though.

As a precaution we could check some of the most popular packages to make sure they have their settings right. I could probably write a script for this.

@gaborbernat
Copy link

gaborbernat commented Mar 10, 2021

What information are you looking out for here? I mean, almost all of @asottile packages will fail essentially, and I'm sure even I have a few lying around (though I don't mind if the CI starts failing). You could in theory get https://hugovk.github.io/top-pypi-packages/, and download each wheel, check if the wheel tag is conflicting with python-reqires. But say 60% fail, what would you do then?

@agronholm
Copy link

Pre-built wheels are not a problem, since wheel won't touch those. The problem lies with those who, for whatever reason, do their builds from source packages.

I would lessen the impact by submitting PRs to projects that have misconfigured their packaging setup.

@pfmoore
Copy link
Member

pfmoore commented Mar 10, 2021

I should also say that right now, there's nothing in any spec that disallows publishing a wheel with (for example) a Python tag of .py37 and a Python-Requires of >=3.8. Such a wheel is uninstallable, but it's not invalid. I see no reason to object if someone were to propose a spec change that tightened this up, but there isn't anything right now, so it's arguably not standards-compliant to reject such wheels. But it seems like a sensible thing to do, if you want to. It's certainly not mandated that you do so, but I'd have little sympathy for anyone who demanded that tools allow uninstallable wheels to be published 🙂

Wheels where the tags and requires-python describe different but overlapping sets of versions is a much harder problem, though (for a start, there's no set of tags that's equivalent to an open-ended requires-python like >= 3.7). I'd say that it's unlikely to be worth doing anything on that in a standards context. But if someone wants to try, I'm happy to listen to the debate and see where it goes 🙂

And that's about all that I think needs to be said from a standards perspective.

@henryiii
Copy link

henryiii commented Mar 10, 2021

To achieve that intent the wheel tag this should map would need to be just py

That would mean you'd need a brand new pip version (and brand new packaging version), which would never work on Python 2 since that's been dropped by both packages. Not against a "py" tag, but it's come too late for this. If it was implemented, it probably would need a new setting; [bdist_wheel] universal=1 is pretty arcane; why do you have to tell bdist_wheel something for a normal, pure Python package?

universal just means "pure python and works on any version"

It's very clearly documented that this means Python 2 + Python 3, from the first release note (wheel 0.10, "Define a new setup.cfg section [wheel]. universal=1 will apply the py2.py3-none-any tag for pure python wheels."), two years before the change linked above (which also still mentions py2.py3). It never was integrated with the Requires-Python, so it always meant "py2.py3" because that's what "universal" meant. Wheel already can detect pure-python; as far as I know, it always has (no Extension present); there's no reason to duplicate this information. It has been clearly documented that a Pure Python wheel is a pure Python wheel that does not support Python 2.

almost all of @asottile packages will fail essentially

He can change one package (setup-cfg-fmt) and they all will be instantly fixed, I expect ;) . And that will help clean up mine too if I have any laying around. :) (And I did have to fix a package about a month ago, so it does happen). That's one reason I really wanted to convince him that this should not be set. I miserably failed, he said it's about pure Python, not about Python 2.

I would lessen the impact by submitting PRs to projects that have misconfigured their packaging setup.

I did. Didn't get a happy response. pre-commit/pre-commit#1832. "general word of advice: don't touch infrastructure or tooling for a project you don't maintain unless prompted by an issue asking explicitly to do so" and "the crusade to remove this flag is not correct and I closed this change because it is not welcome here".

@asottile
Copy link
Contributor

asottile commented Mar 10, 2021

a Python tag of .py37 and a Python-Requires of >=3.8. Such a wheel is uninstallable,

technically that could be installable since py37-none-any means purelib 3.7+ -- if instead that's cp37-cp37m then yes it is uninstallable

@henryiii
Copy link

All older (not c) python3 tags install:

python3.9 -m pip debug -v
...
Compatible tags: 1776
...
  py30-none-macosx_10_4_universal
  cp39-none-any
  py39-none-any
  py3-none-any
  py38-none-any
  py37-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

@agronholm
Copy link

I would also like to deprecate the universal setting already, but what would the proper replacement be? Should wheels be universal by default if there is no python_requires? Probably not. Perhaps then a python_requires that includes Python 2.x?

@henryiii
Copy link

henryiii commented Mar 10, 2021

There's an idea to set the lowest Python supported as the tag via Requires-Python here: pypa/wheel#336 , by the way.

I would also like to deprecate the universal setting already, but what would the proper replacement be?

I don't see a strong point for producing py2.py3. It was a very special situation that hopefully won't be repeated, and most packages are now py3. I would actually rather hope py does get added and turned on in 3-5 years so we don't end up with py3.py4 if there's a Python 4; this should be handled by Requires-Python in the future, rather than tags.

I'm not against getting more packages using the python_requires setting. It's surprisingly rare in cibuildwheel's known users, about 50%.

@pfmoore
Copy link
Member

pfmoore commented Mar 10, 2021

technically that could be installable since py37-none-any means purelib 3.7+ -- if instead that's cp37-cp37m then yes it is uninstallable

Drat, that'll teach me to try to be clever and invent a "better" example (I was originally going with py2 rather than py37, which would also be uninstallable, I believe - although now I'm nervous even about that 🙂) And actually, there's no specification that mandates what compatibility tags a given distribution must say it supports, so from a purely theoretical standpoint, it's even more complex than that...

Let's just go with "it's possible to create wheels that are uninstallable" 😉

@agronholm
Copy link

@asottile Do you still object to the removal of universal=1 in your projects?

@asottile
Copy link
Contributor

@asottile Do you still object to the removal of universal=1 in your projects?

correct

@agronholm
Copy link

Why? When it's clearly wrong to have this in a py3 only project?

@henryiii
Copy link

henryiii commented Mar 10, 2021

I don't see why a user should be expected to make or understand some random universal=1 call to bdist_wheel in setup.cfg. It doesn't translate to PEP 621 configuration, and it is and was never needed to indicate "this is pure python" - the absence of Extensions does that. It was intended from day one (wheel 0.10.0 in 2012, when it was called [wheel] universal=1) to provide a way to go against the original design that Python 2 and Python 3 code could not be run without changes (read the Python 3.0 changelog for the sad statements like that). If we want a "pure_python = true" flag, then it could be added as such, and not as a misused bdist_wheel option; but it's not needed, we don't need a flag that says this is pure Python, and we already have a lot of cruft to just make a simple package (see flit and poetry for attempts to make things shorter/simpler).

For Python 4, we have Requires-Python and the knowledge that people are going to want to write code that runs in both Python 3 and 4 at the same time.

From the 3.0 release notes:

It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; you’d have to use a very contorted coding style, e.g. avoiding print statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the 2to3 translator again, rather than editing the 3.0 version of the source code.

@agronholm
Copy link

The universal=True option was a stopgap measure which was introduced long before requires_python became a thing.

@henryiii
Copy link

bdist_wheel is also becoming an implementation detail, and not something users are expected to call by hand; it's called by pypa/pip and now by pypa/build. Having it in the config seems counter productive to that work.

@agronholm
Copy link

Yes, and setuptools is likely to adopt the bdist_wheel command into itself in the future.

@pfmoore
Copy link
Member

pfmoore commented Mar 10, 2021

It was intended from day one (wheel 0.10.0 in 2012, when it was called [wheel] universal=1) to provide a way to go against the original design that Python 2 and Python 3 code could not be run without changes (read the Python 3.0 changelog for the sad statements like that).

Do you have any documentation that confirms that's how "universal" was intended? I don't recall it being like that - it was originally intended specifically to indicate that a project was "universal" as in "could be run on any Python installation" and as such translated to py2.py3 precisely because that was "any Python installation" at the time. It wasn't "going against the design", although it probably was acknowledging that people were writing cross-version code even though that wasn't how the core devs imagined the transition would work.

The universal=True option was a stopgap measure which was introduced long before requires_python became a thing.

Um, requires_python defines a limited range of Python versions, whereas "universal" says "anything works" (ignoring py4...). They are pretty much opposites of each other, so I'm not sure how universal could be a stopgap until requires_python came along.

It's certainly true that things have changed, though, and the reality nowadays is that "universal" probably isn't a useful designation for real projects, and certainly there are unlikely to be many projects that still work on "all versions of Python". But I'd suggest that we stick to discussion what's useful or worth supporting now, and not try to read too much into the history (or people's recollections of it).

@agronholm
Copy link

Um, requires_python defines a limited range of Python versions, whereas "universal" says "anything works" (ignoring py4...). They are pretty much opposites of each other, so I'm not sure how universal could be a stopgap until requires_python came along.

Well, as you pointed out, it scopes the wheels to Python 2 and 3 only which is not really universal. But realistically, any project has a minimum supported Python version which they should indicate with requires_python. In my opinion this should be a requirement for any packages uploaded to PyPI.

@pfmoore
Copy link
Member

pfmoore commented Mar 10, 2021

it scopes the wheels to Python 2 and 3 only which is not really universal.

It was when we designed the flag. We didn't think sufficiently far ahead, but the intention was "everything".

But realistically, any project has a minimum supported Python version which they should indicate with requires_python

But what I support and what it works on may be two different things. And I may prefer not to block people on older systems from using my code, just because I won't support them. I feel like there's an assumption underlying this that all projects are somehow required to hit a certain bar of "professionalism" here, and to be blunt, that just isn't the case.

In my opinion this should be a requirement for any packages uploaded to PyPI.

How do you plan on fixing all the existing projects that don't do that? I really don't see any benefit in a rule like this that only applies to new uploads.

To give a concrete example, pip's resolver, if faced with the latest version of a package saying that it doesn't support my Python version, will quite happily go back through all the older versions (potentially hundreds of them) looking for one that supports my installation. And ultimately, what it will find won't necessarily be one that supports my version, it's just as likely to find something that's old enough that it was uploaded before the rule requiring requires_python was put in place, and give me some ancient junk that still doesn't work on my system.

We need to remember that PyPI don't have the resource (or, as far as I'm aware, the desire) to be a curated repository of packages, so there's a lot of stuff on there that's of questionable value or is simply abandoned. Imposing new rules without considering how we apply them to existing projects is just going to result in rules that have so many exceptions that people of necessity will ignore them.

Also, what's to stop projects just saying requires_python >1.0? How does mandating the field is present make any practical difference?

Anyway, this is going a long way off topic. I'm going to stop responding to this issue unless explicitly pinged, as I don't think this subthread is going anywhere useful.

I suggest we get back to the original question, of whether twine check should let people know if their python_requires and their wheel tags are inconsistent, and leave the wider infrastructure questions for another time.

@henryiii
Copy link

henryiii commented Mar 10, 2021

Do you have any documentation that confirms that's how "universal" was intended?

The guess about intentions are based on the problem it was trying to solve (memory, the current docs, and old release notes); I just downloaded wheel 0.9.7 and setuptools 0.7.2, and ran bdist_wheel on a simple project. The produced wheel was example_pkg_YOUR_USERNAME_HERE-0.0.1-py27-none-any.whl - this would load on a hypothetical Python 2.8, but not on Python 3. The universal indicated that it would load on Python 3 as well. It also had the nice side effect of allowing it to load on older versions of Python (2.6, for example), but that's not the case anymore. So there was a period between 2012 and 2014 when this also had a nice side effect for Pure Python wheels. This was fixed with wheel 0.23.0 in March, 2014; the default tag became py2 or py3 for pure Python projects. The "universal" flag has only meant "supports Python 2 and 3" since then.

Note that @asottile's projects are Python 3.6+, and have requires_python= >=3.6.1, but he wants to keep the universal=1 and not support Python 2 since: (@asottile please correct me if wrong, based on a message you sent earlier, and I don't really agree so I'm sure I'm not phrasing it well)

  1. He thinks it is a nice piece of documentation that the project is "pure python".
  2. He supports Python 4, and doesn't want to add it back in when Python 4 comes out.

For 1, since setuptools, pip, packaging, and most other Python 3+ pure Python libraries have dropped it, I don't think that's true anymore. It's also very clearly documented everywhere it shows up in wheel and packaging.python.org's guides that it means Python 2 + 3, and pure Python wheels do not need universal=1.

For Python 4, I don't think we could reuse [bdist_wheel] universal=1. It clearly currently means py2.py3, and will there really be a py2.py3.py4? Where does it end? That's why Requires-Python is better; every package with any code in it to speak of has some minimum Python version - it's not Universal. Also, the meaning of Universal should change over time, but packaging shouldn't change over time. Even 2.7, 3.5+ isn't really "universal", which is also why python_requires=... is better. So it's not really currently useful, and it's probably not useful in the future.

For macOS, "Universal" meant PowerPC+Intel. "Fat" meant Intel32+Intel64. And now "Universal2" means Intel+AppleSilicon. I think we could avoid another special flag to bdist_wheel by using Requires-Python, but say we didn't; it probably would have to be universal2=True or just the tag, py=py3.py4, etc. Actually, py36.py41 or something like that would likely be even better and more readable for users. That's what pypa/wheel#336 wants to do.

@henryiii
Copy link

Note, I do know a few projects that "secretly" supported a version but set the requires one higher. It's not really that helpful of a practice, I think, but it is a rare case where they want to have a py2.py3 tag, since it does work on Python 2, so in a pinch it can be used (though pip won't resolve it, so it's not that useful). But this is a "special" case (and not @asottile's case at all). If it's unofficial support and requires a workaround to install anyway, changing the Python tag isn't that hard.

@henryiii
Copy link

henryiii commented Mar 11, 2021

How about this concrete proposal:

  • No Requires-Python, no [bdist_wheel] universal=1: produce py3 wheels, as now.
  • No Requires-Python, has [bdist_wheel] universal=1: produce py2.py3 wheels, as now. No warning, fully allowed for now.
  • Requires-Python, no [bdist_wheel] universal=1: Use the smallest allowed tag for each valid Python major version.
    • >=2.7 would produce py27.py3.
    • >=2.6, !=3.0.*, !=3.1.* would produce py26.py32.
    • >=3.6 would produce py36.
    • When Python 4 comes out, this starts including Python 4 if allowed by Requires-Python, just like 2/3.
  • Python-Requires, has [bdist_wheel] universal=1: produce a warning from wheel, maybe eventually an error (after Python 2 is dropped from wheel). This is logically over constrained - it shouldn't universally work on all Pythons and be limited to some Pythons. When producing a warning, it still produces py2.py3 wheels for backward compatibility.

This would mostly be implemented through pypa/wheel#336. This has a small consequences to the point of this thread:

  • Twine could start warning if Requires-Python is set to 3+ and universal=1 was given. It would be invalid to warn for 2+ and universal=1 until the feature described above was implemented and common.

Edit: added missing behavior when both are given.

@asottile
Copy link
Contributor

imo attempting to match a wheel tag to python_requires is kind of a lost cause, what would you do for python_requires >= 3.6.1? (py36 isn't quite right, and py37 definitely isn't correct, and py3 has the same "problems" that py2.py3 has)

@henryiii
Copy link

henryiii commented Mar 11, 2021

Three possibilities.
a) use 99 as the patch; patches can't go above about 18. This worked on all cibuildwheel's users, and has been shipped and has been working for a while (cibuildwheel 1.9 IIRC). This only fails if an upper patch bound is made, which is pretty invalid. This is also clearly suggested in the issue linked above if you read it.
b) Check against a list of 20 patch versions for each Python, and see if they match. If any match, that once can be used. A bit slower.
c) Compute the cross section analytically; if any X.Y.* is allowed, then use that XY. More trouble, but probably doable, and should be as fast as a. A SpecifierSet is pretty simple.

I'm also personally fine with py3, this was based partially on the linked issue wanting to make this more descriptive; there's nothing wrong with a py3 wheel that works on py36+ IMO. py3 means it works on some versions of Python 3; it almost never means it works on 3.0/3.1/3.2. But a py2.py3 file that does not support any version of Python 2 is wrong. Also, having Tag: py2-none-any in .dist-info/WHEEL is wrong too.

Similarly, py36 should work on some version of Python 3.6. I'd maybe even state it must work on future versions of Python 3.6 (well, if there were any).

CERN still has running Scientific Linux 6 machines that have pip 8, FFI. It's not truly dead; I'm very glad at least pip correctly removed py2 when updating to 21. 7% of Python 2 users have pip < 9. <unrelated rant> Though nothing like Pip 9. Ubuntu 18.04 comes with Pip 9! Even on Python 3.8! CentOS 8 comes with Pip 9! Pip 9 can't load manylinux2010+! </unrelated rant>

@henryiii
Copy link

These sorts of discussions really should be made on the wheel issue, not here.

@pradyunsg
Copy link
Member

IMO twine printing warning when there's some sort of obvious mismatch like the valid-but-uninstallable wheel example that @pfmoore gave above, or a wheel that's tagged for Python 2 but can't be installed there due to python_requires is a good idea.


It's probably a good idea to check if the last version of pip that supported Python 2 does not work with py, because if it does, then there's not much discussion to be had here. :)


imo attempting to match a wheel tag to python_requires is kind of a lost cause, what would you do for python_requires >= 3.6.1?

FWIW, I think this issue's OP isn't to suggest that we should try to have them match precisely though. Rather, it's about being tagged for a Python version that the package can't be installed on. And this whole thing about inferring tags should likely move to the wheel issue tracker.

FWIW, I'd like to put the py tag on this example TBH. Because we don't have that yet, it'd need to be py3 instead (which is fine until we have to figure out what to do for Python 4, and continuing to put py2 here isn't going to help with that problem anyway).

We can add py in before Python 3/4 though, as @henryiii suggests, but that's probably better for discuss.python.org.

@bhrutledge
Copy link
Contributor

👋 (relatively new) Twine maintainer here. It's been interesting to watch this discussion, but I've only skimmed it. Can somebody write (or point to) a concise proposal for an enhancement to Twine that has (or can get) the support of the folks involved?

@sigmavirus24
Copy link
Member

@bhrutledge I'd say there does not exist a proposal which everyone who has been involved in this discussion would be supportive of. I'll try to summarize:

  • Today when you indicate that the wheel you're building is universal, it adds py2.py3 because at the time of the implementation that was what "universal" meant (works on 2 and 3)
  • Pip versions for years have relied on that as a hint as to which wheel should be installed on a given version of Python
  • "universal" semantically means any version of Python
  • Metadata has grown the "python-requires" field which allows you to have a "universal" wheel so long as "universal" also matches the python-requires. (Many folks use python-requires to indicate the minimum version they support of Python) That seems to be contradictory though given that pypa/wheel tags "universal" wheels with py2.py3 and python-requires can have > 3.5

I think the best we could do is have a check that can never be considered an error because the confidence in that check would be so low that is also only run when provided a flag that warns if you have requires-python and the wheel is tagged py2.py3. That said, we'd be wading into a minefield by doing so and I'm not sure it's worth the effort.


All of the above is to say: There's almost no alignment on what any of this means (the current maintainers of various pypa projects seem to have different interpretations of things compared to the PEPs when they were written) and tools have grown behaviour based on what was available. Twine taking any action here seems like it would result in endless churn and heartburn for us.


Finally as a total aside, it seems all members of PyPA can tag issues on this project despite not being maintainers and thus can give an issue a sense of being accepted as a feature/enhancement. That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.

@webknjaz
Copy link
Member Author

Finally as a total aside, it seems all members of PyPA can tag issues on this project despite not being maintainers and thus can give an issue a sense of being accepted as a feature/enhancement. That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.

Oh, sorry about that. I believe that being in the "Gardeners" team grants this right. That team is meant to give other PyPA folks the ability to do some harmless housekeeping but it seems like not everybody's aware of this.

Twine taking any action here seems like it would result in endless churn and heartburn for us.

FWIW I think I'm okay with a "no" if that's what people here think is best. I don't have a strong opinion and only raised this issue because I overlooked this in my projects and didn't even notice until Bernát told me.

I think the best we could do is have a check that can never be considered an error because the confidence in that check would be so low that is also only run when provided a flag that warns if you have requires-python and the wheel is tagged py2.py3.

This solution sounds compelling. I'd definitely +1 it if that's something others can agree on. It is good to have an optionally enabled warning that could become an error with an explicit --strict (#430).

@pfmoore
Copy link
Member

pfmoore commented Mar 14, 2021

Pip versions for years have relied on that as a hint as to which wheel should be installed on a given version of Python

To clarify this point slightly, the wheel tags say which Python version(s) a wheel is compatible with, so pip ignores any wheel that has tags saying it isn't compatible with the target system. This is not really a "hint", it's mandated behaviour - you must not install a wheel whose tags declare that it's incompatible.

The only way we use tags to choose which wheel to install, is in the sense that we choose the "most compatible" wheel where multiple otherwise-equivalent wheels are available. "Universal" (whatever that means in practice) will likely be low on that scale, by nature of what it means to be "universal". Again, that's mandated by the standard so all installers will do this, not just pip.

That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.

If my comments have had that effect, I'm sorry. I have no authority here and my comments are only intended to be informative. I only got involved because @gaborbernat pinged me for a view on what the standards said, and I've tried my best to confine my comments to clarifications on what the standard-mandated behaviour is (which frankly, is only peripheral, as the issue is about a check, which is not something the standards have anything to say on).

FWIW, I entirely support the project maintainers having the final say, and I do not want any comments I've made here to be seen as exerting any pressure on them.

@henryiii
Copy link

henryiii commented Mar 14, 2021

The discussion here got a little far afield, but the focus is not on universal=1, which is a custom, non-standards controlled flag directly to the bdist_wheel command, part of wheel.

The thing that twine check could (and personally, I think should) to is make sure that the Requires-Python metadata slot and the Python tags (which are both standards controlled) do not conflict. Something with > 3.5 should not have Tag: py2-none-any. I think that could easily be a warning from twine check. You don't have to follow warnings from twine check.

Whether universal=1 can produce a non-Python 2 tagged wheel is something to take up with wheel, not here. But until something like that is added, Python 3 only wheels from wheel that have universal=1 set are producing poorly tagged wheels that should be warned about. Using Requires-Python instead of relying on universal=1 to bdist_wheel, introducing a py tag, all sounds like things that could be considered, but out of the scope of twine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Issues we can't or shouldn't get to yet feature request question Discussion/decision needed from maintainers
Projects
Better `check`
  
To do
Development

No branches or pull requests

9 participants