Hash-checking mode breaks after new distributions are added

[di]

What's the problem this feature will solve?
Currently, it's possible to add new distributions to an existing release on PyPI at any point after the release was initially made.

Occasionally when this happens, the new distribution is preferred by pip over previous distributions. For example, initially there is only a source distribution, then 1 week later the maintainer adds a built distribution.

For users who have included hashes in their requirements files, this occasionally leads to breakage (THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.) because pip is preferring a distribution that didn't exist when the user added the hashes, and (correctly) refusing to install it.

The current behavior is roughly:

• find the most ideal distribution
• check if that distribution's hash is in the list of hashes
• install if yes, fail if no

Describe the solution you'd like
The behavior of pip could be this:

• find any installable distributions
• sort them from most to least ideal (i.e., platform-specific wheels before source distributions)
• install the first one that has a matching hash
• fail if it can't install any of them

This would maintain the same security that hash-checking mode provides, while avoiding the potential for breakage when new distributions are added to a release.

Alternative Solutions
Some alternate things we could do:

• PyPI could require that maintainers upload their distributions all at once
  • This would add an additional maintenance burden for the maintainers who build distributions on a variety of different platforms
• PyPI could require that maintainers upload their distributions within some arbitrary window
  • This has all the same problems we currently have, just in a narrower window of time
• We could tell users that they need to use --only-binary and --no-binary flags
  • This is an additional burden to users
  • This could still break if the maintainer deletes the .tar.gz source distribution and uploads a .zip source distribution (or vice versa)
  • This could still break if the maintainer adds a universal wheel when there used to be only a platform-specific wheel.
• We could tell users that they need to specify direct links to distributions on pythonhosted.org instead
  • This is an additional burden to users
  • This doesn't always work if the dependencies need to be installable across multiple platforms

Additional context
Some additional discussion at pypa/packaging.python.org#564.

[dstufft]

I think it's a good idea to do the solution described here, to limit our installation to only things we have hashes for. Ideally I think we'd print a warning of some kind if we're not installing the preferred file due to it not having a hash listed though.

[uranusjr]

I believe this is the same as #3634.

[bennuttall]

I opened a similar issue (which I've now closed in favour of this) #6395 describing this problem but for distributions on additional indexes. We had some reports of issues from people trying to install certbot on Raspberry Pi and picking up wheels from piwheels.org but having a hash-mismatch.

Also see related but different #6394 - proposal to add a flag to remove additional indexes.

[alexbecker]

I think #6464 solves this and addresses @dstufft's concern by logging a warning when a file is skipped due to not having hashes. Before I polish it up (add tests, fix 1 test that broke for reasons orthogonal to correctness), can I get approval-in-principal on the approach?

[indygreg]

On quick glance, the behavior in #6464 - ignoring artifacts for which we don't have a pinned hash - seems like a reasonable solution to this issue! Thank you for coding it up!

[cjerdonek]

This has now been addressed by PR #6699. In a subsequent PR, I'll do what @dstufft suggested above (I'll file a separate issue about that in a bit):

Ideally I think we'd print a warning of some kind if we're not installing the preferred file due to it not having a hash listed though.

There is also a pending PR #6714 that adds some additional debug logging.

[di]

Thank you @cjerdonek!