Support PEP 658 metadata file hosting

[thejcannon]

See PEP 658.

But TL;DR: Now indexes can opt-in to also hosting a package's metadata alongside the package so that package managers don't have to download the package (which in the case of PyTorch can be >GB) to simply resolve metadata (e.g. "what packages do you depend on?").

The support looks roughly like:

• In the index HTML, the anchor tag has a data-core-metadata attribute.
  • If the attribute exists that means you support metadata hosting
  • The attribute value could be true to say "yup, I support this"
  • The attribute value could alternatively be the algorithm+hash of the metadata to allow clients to avoid re-downloading. E.g. sha256=deadbeef
• Your index supports hosting the metadata file at <whl name>.metadata.

As an example from PyPI. If you go to https://pypi.org/simple/requests/ and look at the HTML you'll find:

<a href="https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl#sha256=<package hash>" 
    data-requires-python=">=3.7" 
    data-dist-info-metadata="sha256=<metadata hash>" 
    data-core-metadata="sha256=<metadata hash>"
>requests-2.31.0-py3-none-any.whl</a>

(Note that there's two attributes with similar names due to a nasty bug in pip requiring a rename of the attribute. See PEP 714. I assume PyPI is simply just exposing both (something this index could do as well)

Then, since data-core-metadata exists, voilà! https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata

Note that this is related-to, but not quite the same as #1347, which is specifically for sha256 PEP 503 support.

[thejcannon]

Oh and the juicy part, if you use bleeding-edge pip (like clone the repo and build it, I think the next release is sometime this month though) and try to run a command that doesn't need to install requests, just requires metadata (as is common for locking operations) you'll see it run much faster comparatively.

Something like pip install --dry-run --quiet --report - --no-deps requests==2.31.0.

Note that PyPI is looking into backporting the metadata hosting to older wheels but hasn't yet (which is why I decided to use --no-deps).

[akx]

+1 for this!

Related (linking here so those issues get backlinks):

• Pipenv PEP 503 Improvement: Pipenv downloads PyTorch for all versions of Python, grabbing 16GB of data instead of just 1.7GB. pypa/pipenv#4963
• Multi Platform support for Pipfile.lock pypa/pipenv#5130
• Pip packaging and publishing improvements in pytorch wheels for better integration with poetry pytorch#76557