Pip packaging and publishing improvements in pytorch wheels for better integration with poetry

[suneeta-mall]

🐛 Describe the bug

Poerty is a great build toolchain for python applications that lean on pip packaging.

Please see a detailed thread here python-poetry/poetry#4231 (comment) on why the integration of applications written using PyTorch eco-system fails. As mentioned in the follow-up reply from @abn here python-poetry/poetry#4231 (comment) that some improvements in our PyTorch packages are published and indexed can greatly improve the experience and integration. To highlight the requests:

1. The lack of file hashes being exposed in the repository means that Poetry has to download each artifact unnecessarily. As per PEP 503. this can be appended to the file URL as #=. This can be done easily by the index admins I believe.
2. While fixing (1) will reduce the number of required downloads, files still will be downloaded again when there is a need to inspect metadata (ie. Requires-Dist, etc. from the wheel). This can be avoided too if PEP 658 gets implemented. I suspect this will need to also be supported by Poetry as well.
3. Would be great if requesting a non-existant resource returns a 404 instead of 403. If I understand correctly, this is because they have not set s3:ListBucket permission for public users (assuming they use S3 for this).

I am not 100% sure this ticket is the best place on the main torch repo but I did not find any CI-specific repo either in pytorc. Please help direct if I got the main repo to report this wrong.

Versions

Has been tried with multiple versions including 1.11

cc @seemethere @malfet @osalpekar @atalman @ezyang

[matteius]

This also is an issue for pipenv for the same reasons -- I have been considering an improvement to pipenv where we would look for the same style pypi json API response for retrieving packages hashes from the index, before falling back to downloading each of the wheels individually to obtain the hashes. I came here to post a similar issue, but I found this one--perhaps our goals are similar enough that we can leverage a similar solution for this? Otherwise I could open a new ticket as well if that is preferred.

Note: I have the other night I began prototyping what such an API response looks like in the contetext of pypiserver which this PR might give you an idea of the kind of json pipenv would be looking to consume: pypiserver/pypiserver#440

[Damowerko]

Now Poetry 1.2 can handle PEP 503 repositories that implement PEP 658. Could the PyTorch PEP 503 repository implement this? And would it resolve the issue?

[matteius]

I saw @atalman cut the last release this past week so I am tagging him on this issue -- having a way to validate the package hashes without downloading all of the wheels for that version would be incredibly useful to both poetry and pipenv. I realize there are over 5000 open issues here, so my hope is that by surfacing this problem to the right person it would be mutually beneficial to your bandwidth consumed and to our users.

[juliusfrost]

@suneeta-mall It looks like this code that generates the index is managed in here: https://github.com/pytorch/builder/tree/main/s3_management

[ralbertazzi]

Note how point 3, although in general desirable, should not be strictly needed anymore with the release of Poetry 1.5 and the support of explicit sources: python-poetry/poetry#6409 (comment)

[ralbertazzi]

PEP 658 is also live on pypi (https://discuss.python.org/t/pep-658-is-now-live-on-pypi/26693), but Poetry still has to support it. So right now the only high priority point is lack of file hashes.

[vit-zikmund]

Point 1. seems done! 😱 The torch wheels I saw have hashes now! 🎉

[matteius]

@vit-zikmund some of them do -- unfortunately, in my test example it resolves to https://download.pytorch.org/whl/cu118 torch torch-2.0.1 which doesn't have them in the link, but some wheels do now 🤔

[malfet]

@matteius script that repopulates S3 objects with hashes is still not finished (as there are a lots of historical wheels), but it should be done hopefully soon.

[matteius]

@malfet This is awesome, I just tried it out and pipenv is so much faster to lock pytorch now 🚀

[vit-zikmund]

@matteius poetry still sucks at locking tho (downloads all platforms * pythons before success). I wonder what pipenv does differently... or.. didin't you mean "installing from a lockfile" (because that would make a lot more sense, otherwise all the lockfile-producing logic needs to read the packages metadata, at least for dependencies, which is still not readily available without downloading the full thing).

[dimbleby]

progress seems to have been reversed eg torch 2.1.1 wheels at https://download.pytorch.org/whl/torch/ do have hashes on their URLs but torch 2.1.2 wheels do not

[atalman]

@dimbleby I see you right. This is the issue you are talking about ?

torch 2.1.1 following html is generated:

<a href="/whl/cpu/torch-2.1.1-cp39-none-macosx_10_9_x86_64.whl#sha256=715b50d8c1de5da5524a68287eb000f73e026e74d5f6b12bc450ef6995fcf5f9">torch-2.1.1-cp39-none-macosx_10_9_x86_64.whl</a>

torch 2.1.2:

<a href="/whl/cpu/torch-2.1.2%2Bcpu-cp310-cp310-linux_x86_64.whl">torch-2.1.2+cpu-cp310-cp310-linux_x86_64.whl</a>

will look into this issue

[dimbleby]

seems like the hashes come and go, at time of writing the links from https://download.pytorch.org/whl/torch/ include them on most distributions but not on torch 2.2.1 packages.

[malfet]

@atalman can you please make sure to add hash generation to the release pipeline runbook?

In the meantime, run the script manually to re-populate checksum

[PyroGenesis]

@atalman Hi, again 2.2.1+cu121 has hashes but 2.2.2+cu121 does not.