building docker image fails on M1

[fritjof-b]

When building the docker image on Mac M1 with command docker build --no-cache --progress="plain" -t baler:arm . it fails.

As far as we know this is because we're using PyTorch 1.13.1, which pulls in Nvidia Cuda toolkit. This isn't natively available for ARM processors. The build fails when poetry is looking for a compatible version.

Related to:

• Poetry unable to find installation candidates for Nvidia c11 dependencies python-poetry/poetry#6939
• https://developer.nvidia.com/nvidia-cuda-toolkit-11_7_0-developer-tools-mac-hosts

#11 9.370   • Installing nvidia-cublas-cu11 (11.10.3.66)
#11 9.434 
#11 9.447   RuntimeError
#11 9.447 
#11 9.447   Unable to find installation candidates for nvidia-cublas-cu11 (11.10.3.66)
#11 9.448 
#11 9.448   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:105 in choose_for
#11 9.454       101│ 
#11 9.458       102│             links.append(link)
#11 9.458       103│ 
#11 9.458       104│         if not links:
#11 9.458     → 105│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#11 9.465       106│ 
#11 9.465       107│         # Get the best link
#11 9.465       108│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#11 9.465       109│ 
#11 9.465 
#11 9.465   • Installing nvidia-cuda-nvrtc-cu11 (11.7.99)
#11 9.481 
#11 9.481   RuntimeError
#11 9.481 
#11 9.481   Unable to find installation candidates for nvidia-cuda-nvrtc-cu11 (11.7.99)
#11 9.481 
#11 9.481   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:105 in choose_for
#11 9.486       101│ 
#11 9.486       102│             links.append(link)
#11 9.486       103│ 
#11 9.486       104│         if not links:
#11 9.486     → 105│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#11 9.486       106│ 
#11 9.486       107│         # Get the best link
#11 9.486       108│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#11 9.487       109│ 
#11 9.487 
#11 9.487   • Installing nvidia-cuda-runtime-cu11 (11.7.99)
#11 9.501 
#11 9.501   RuntimeError
#11 9.501 
#11 9.501   Unable to find installation candidates for nvidia-cuda-runtime-cu11 (11.7.99)
#11 9.501 
#11 9.501   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:105 in choose_for
#11 9.506       101│ 
#11 9.506       102│             links.append(link)
#11 9.507       103│ 
#11 9.507       104│         if not links:
#11 9.507     → 105│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#11 9.507       106│ 
#11 9.507       107│         # Get the best link
#11 9.507       108│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#11 9.507       109│ 
#11 9.507 
#11 9.507   • Installing nvidia-cudnn-cu11 (8.5.0.96)
#11 9.523 
#11 9.523   RuntimeError
#11 9.523 
#11 9.523   Unable to find installation candidates for nvidia-cudnn-cu11 (8.5.0.96)
#11 9.523 
#11 9.523   at /usr/local/lib/python3.8/site-packages/poetry/installation/chooser.py:105 in choose_for
#11 9.528       101│ 
#11 9.528       102│             links.append(link)
#11 9.528       103│ 
#11 9.529       104│         if not links:
#11 9.529     → 105│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#11 9.529       106│ 
#11 9.529       107│         # Get the best link
#11 9.529       108│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#11 9.529       109│ 
#11 9.529 
#11 9.529   • Installing pandas (1.5.3)
#11 9.537   • Installing pillow (9.4.0)
#11 9.656   • Installing pluggy (1.0.0)
#11 9.672   • Installing pyparsing (3.0.9)
#11 9.719   • Installing scipy (1.9.3)
#11 9.888   • Installing threadpoolctl (3.1.0)
#11 10.10   • Installing tomli (2.0.1)
#11 10.38   • Installing uproot3-methods (0.10.1)
#11 ERROR: executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi]: exit code: 1
------
 > [python-base 6/8] RUN poetry install --no-interaction --no-ansi:
------
executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi]: exit code: 1

[fritjof-b]

I've pushed a Dockerfile that successfully builds and runs on my setup (M1) on branch related to this issue.

This was done by excluding CUDA libraries from PyTorch, this reduced the file size from approx 5.7 GB to 590 MB. I don't know the exact implications of this, but since we're not training on GPU on Mac anyway. Does it matter for the project? @exook @OliverWoolland

[OliverWoolland]

This sounds like a good idea for the main Dockerfile too! I might be being thick though - I can't see the change which excludes CUDA libraries from the new Dockerfile?

[OliverWoolland]

Ah is it the --only-root flag for poetry?

[fritjof-b]

Yes, exactly. I also changed the base image to ARM64 Python (not sure what this would imply, but seemed like good practice). I also added updates to system as well as installing poetry through curl.

Regarding the CUDA libraries, I'm not sure -- aren't they needed for the GPU acceleration for people with Nvidia GPUs (which some might have). Nonetheless, saves a ton of space... :-)

[OliverWoolland]

I think a main image which is cpu only is not unreasonable! Then a GPU tag which includes the heavy stuff?

I spotted the change to the poetry install command - I guess it makes it more independent of the base image used? Does away with the need for pip in the first image too. Nice