Broken Docker Image on dockerhub

[lharri73]

🐛 Describe the bug

It appears that the docker image on dockerhub for 2.3.0 cuda11.8 & cuda12.1, both runtime and devel are all malformed.

1. They target arm64 instead of amd64 like all previous images
2. The copy from /opt/conda only copied ~300Mb where it normaly copies several Gb.

Versions

N/A. Image will not run.

cc @ezyang @gchanan @zou3519 @kadeng

[malfet]

Hmm, indeed it is the case:

$ docker run --rm -it pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime bash
Unable to find image 'pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime' locally
2.3.0-cuda12.1-cudnn8-runtime: Pulling from pytorch/pytorch
Digest: sha256:cc14d1be87739710ca4e14c344e5d336b4dafde40df1a02cc5ac5c265301868c
Status: Downloaded newer image for pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested

exec /usr/bin/bash: no such file or directory

and

$ docker run --rm -it pytorch/pytorch:2.3.0-cuda12.1-cudnn8-devel bash
Unable to find image 'pytorch/pytorch:2.3.0-cuda12.1-cudnn8-devel' locally
2.3.0-cuda12.1-cudnn8-devel: Pulling from pytorch/pytorch
Digest: sha256:0822df0b146549df1f487e30613e4aacf2976185587028866aa98701ea2e5ca8
Status: Downloaded newer image for pytorch/pytorch:2.3.0-cuda12.1-cudnn8-devel
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested
exec /opt/nvidia/nvidia_entrypoint.sh: no such file or directory

@atalman can you please fix it ASAP and let's try to figure out later how it happened?

[Edit] Most likely culprit is this guy: #115949

[janvdp]

Hi @lharri73, as a temporary workaround I'm using: "ghcr.io/pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime"

Maybe it helps...

[atalman]

@janvdp, @malfet the images in ghcr.io and pytorch/pytorch should be exactly the same here is the log:

pytorch/pytorch           2.3.0-cuda11.8-cudnn8-runtime   3578e171db9e   4 days ago     1.18GB
ghcr.io/pytorch/pytorch   2.3.0-cuda11.8-cudnn8-runtime   3578e171db9e   4 days ago     1.18GB
ghcr.io/pytorch/pytorch   2.3.0-cuda11.8-cudnn8-devel     d0edb1392485   4 days ago     9.05GB
pytorch/pytorch           2.3.0-cuda11.8-cudnn8-devel     d0edb1392485   4 days ago     9.05GB
ghcr.io/pytorch/pytorch   2.3.0-cuda12.1-cudnn8-runtime   994d45086c44   4 days ago     1.18GB
pytorch/pytorch           2.3.0-cuda12.1-cudnn8-runtime   994d45086c44   4 days ago     1.18GB
pytorch/pytorch           2.3.0-cuda12.1-cudnn8-devel     c270f91fbe3e   4 days ago     9.24GB
ghcr.io/pytorch/pytorch   2.3.0-cuda12.1-cudnn8-devel     c270f91fbe3e   4 days ago     9.24GB

Here is the validation workflow for these images:
https://github.com/pytorch/builder/actions/runs/8821461020/job/24217375234#step:11:57

Please note failure you see in the validation workflow is caused by this issue, still open:
#116696

Issue is due to ghcr.io/pytorch/pytorch contains both images arm64 and amd64:

For release 2.3 I uploaded arm64 images. Will upload amd64 image now to fix this issue.

[atalman]

Amd64 images uploaded:

[malfet]

@atalman just curious, what's inside 2.3.0-cuda11.8-cudnn8-runtime arm64 image? I assume now CUDA components are bundled with it, are there? In that case, why is there cuda-11.8 in its tag name?

[atalman]

@malfet Looks like this is an error. arm64 images should only be: ghcr.io/pytorch/pytorch:2.3.0-runtime

Since they build without CUDA support:

docker run --rm -it ghcr.io/pytorch/pytorch:2.3.0-cuda11.8-cudnn8-runtime bash
root@a46ecf7eefa2:/workspace# python
Python 3.10.14 (main, Mar 21 2024, 16:18:23) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.cuda.is_available())
False
>>> 

Created an issue to fix this: pytorch/builder#1806