Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to ubuntu 20.04 #3393

Merged
merged 6 commits into from Feb 3, 2022
Merged

Upgrade to ubuntu 20.04 #3393

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9333018
Move to Ubuntu 20.04
EnricoMi Feb 1, 2022
21c8f68
Use Ubuntu 18.04 along with Python 3.7
EnricoMi Feb 1, 2022
d5741ad
Use Ubuntu 18.04 along with CUDA 10.x
EnricoMi Feb 1, 2022
7bf3715
Move to Python 3.8 in released images
EnricoMi Feb 1, 2022
2d5160c
Run apt-get with DEBIAN_FRONTEND=noninteractive
EnricoMi Feb 2, 2022
11d3bce
Set docker-compose timeout to 5m
EnricoMi Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 5 additions & 2 deletions Dockerfile.test.cpu
View file
Open in desktop
@@ -1,9 +1,9 @@
ARG UBUNTU_VERSION=18.04
ARG UBUNTU_VERSION=20.04
FROM ubuntu:${UBUNTU_VERSION}

# Arguments for the build. UBUNTU_VERSION needs to be repeated because
# the first usage only applies to the FROM tag.
ARG UBUNTU_VERSION=18.04
ARG UBUNTU_VERSION=20.04
ARG MPI_KIND=OpenMPI
ARG PYTHON_VERSION=3.6
ARG GPP_VERSION=7
Expand All @@ -22,6 +22,9 @@ ARG SPARK_PACKAGE=spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
ARG CCL_PACKAGE=master
ARG HOROVOD_BUILD_FLAGS=""

# to avoid interaction with apt-get
ENV DEBIAN_FRONTEND=noninteractive

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]

Expand Down
7 changes: 5 additions & 2 deletions Dockerfile.test.gpu
View file
Open in desktop
@@ -1,9 +1,9 @@
ARG CUDA_DOCKER_VERSION=10.0-devel-ubuntu18.04
ARG CUDA_DOCKER_VERSION=10.0-devel-ubuntu20.04
FROM nvidia/cuda:${CUDA_DOCKER_VERSION}

# Arguments for the build. CUDA_DOCKER_VERSION needs to be repeated because
# the first usage only applies to the FROM tag.
ARG CUDA_DOCKER_VERSION=10.0-devel-ubuntu18.04
ARG CUDA_DOCKER_VERSION=10.0-devel-ubuntu20.04
ARG CUDNN_VERSION=7.6.0.64-1+cuda10.0
ARG NCCL_VERSION_OVERRIDE=2.4.7-1+cuda10.0
ARG MPI_KIND=OpenMPI
Expand All @@ -23,6 +23,9 @@ ARG SPARK_PACKAGE=spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
ARG HOROVOD_BUILD_FLAGS="HOROVOD_GPU_OPERATIONS=NCCL"
ARG HOROVOD_MIXED_INSTALL=0

# to avoid interaction with apt-get
ENV DEBIAN_FRONTEND=noninteractive

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]

Expand Down
21 changes: 16 additions & 5 deletions docker-compose.test.yml
View file
Open in desktop
Expand Up @@ -5,7 +5,7 @@ services:
context: .
dockerfile: Dockerfile.test.cpu
args:
UBUNTU_VERSION: 18.04
UBUNTU_VERSION: 20.04
GPP_VERSION: 7
MPI_KIND: None
PYTHON_VERSION: 3.8
Expand All @@ -24,6 +24,8 @@ services:
# our baseline first
test-cpu-gloo-py3_8-tf2_7_0-keras2_7_0-torch1_10_1-mxnet1_9_0-pyspark3_2_0:
extends: test-cpu-base

# permute MPI kinds
test-cpu-mpich-py3_8-tf2_7_0-keras2_7_0-torch1_10_1-mxnet1_9_0-pyspark3_2_0:
extends: test-cpu-base
build:
Expand Down Expand Up @@ -55,6 +57,9 @@ services:
extends: test-cpu-base
build:
args:
# Tensorflow 1.15.5 is only available for Python 3.7
# Python 3.7 is only available on Ubuntu 18.04
UBUNTU_VERSION: 18.04
PYTHON_VERSION: 3.7
# there is no tensorflow-cpu>1.15.0, so we use tensorflow==1.15.5
TENSORFLOW_PACKAGE: tensorflow==1.15.5
Expand Down Expand Up @@ -97,6 +102,9 @@ services:
extends: test-cpu-base
build:
args:
# Tensorflow 1.15.5 is only available for Python 3.7
# Python 3.7 is only available on Ubuntu 18.04
UBUNTU_VERSION: 18.04
PYTHON_VERSION: 3.7
PYSPARK_PACKAGE: pyspark==2.4.8
SPARK_PACKAGE: spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz
Expand Down Expand Up @@ -136,6 +144,8 @@ services:
extends: test-gpu-base
build:
args:
# Tensorflow 1.15.5 is only available for Python 3.7
# Python 3.7 is only available on Ubuntu 18.04
CUDA_DOCKER_VERSION: 10.0-devel-ubuntu18.04
CUDNN_VERSION: 7.6.5.32-1+cuda10.1
NCCL_VERSION_OVERRIDE: 2.7.8-1+cuda10.1
Expand All @@ -151,6 +161,7 @@ services:
extends: test-gpu-base
build:
args:
# CUDA 10.x devel image is only available with Ubuntu 18.04
CUDA_DOCKER_VERSION: 10.1-devel-ubuntu18.04
CUDNN_VERSION: 7.6.5.32-1+cuda10.1
NCCL_VERSION_OVERRIDE: 2.7.8-1+cuda10.1
Expand All @@ -164,7 +175,7 @@ services:
extends: test-gpu-base
build:
args:
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu18.04
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu20.04
CUDNN_VERSION: 8.1.1.33-1+cuda11.2
NCCL_VERSION_OVERRIDE: 2.8.4-1+cuda11.2
TENSORFLOW_PACKAGE: tensorflow-gpu==2.6.2
Expand All @@ -177,7 +188,7 @@ services:
extends: test-gpu-base
build:
args:
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu18.04
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu20.04
CUDNN_VERSION: 8.1.1.33-1+cuda11.2
NCCL_VERSION_OVERRIDE: 2.8.4-1+cuda11.2
MPI_KIND: OpenMPI
Expand All @@ -191,7 +202,7 @@ services:
extends: test-gpu-base
build:
args:
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu18.04
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu20.04
CUDNN_VERSION: 8.1.1.33-1+cuda11.2
NCCL_VERSION_OVERRIDE: 2.8.4-1+cuda11.2
TENSORFLOW_PACKAGE: tf-nightly-gpu
Expand All @@ -205,7 +216,7 @@ services:
extends: test-gpu-base
build:
args:
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu18.04
CUDA_DOCKER_VERSION: 11.2.2-devel-ubuntu20.04
CUDNN_VERSION: 8.1.1.33-1+cuda11.2
NCCL_VERSION_OVERRIDE: 2.8.4-1+cuda11.2
MPI_KIND: OpenMPI
Expand Down
5 changes: 2 additions & 3 deletions docker/horovod-cpu/Dockerfile
View file
Open in desktop
@@ -1,4 +1,4 @@
FROM ubuntu:18.04
FROM ubuntu:20.04

# TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully
ARG TENSORFLOW_VERSION=2.5.0
Expand All @@ -10,8 +10,7 @@ ARG MXNET_VERSION=1.8.0.post0
ARG PYSPARK_PACKAGE=pyspark==3.1.1
ARG SPARK_PACKAGE=spark-3.1.1/spark-3.1.1-bin-hadoop2.7.tgz

# Python 3.7 is supported by Ubuntu Bionic out of the box
ARG PYTHON_VERSION=3.7
ARG PYTHON_VERSION=3.8

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-cu"]
Expand Down
5 changes: 2 additions & 3 deletions docker/horovod/Dockerfile
View file
Open in desktop
@@ -1,4 +1,4 @@
ARG CUDA_DOCKER_VERSION=11.2.2-devel-ubuntu18.04
ARG CUDA_DOCKER_VERSION=11.2.2-devel-ubuntu20.04
FROM nvidia/cuda:${CUDA_DOCKER_VERSION}

# Arguments for the build. CUDA_DOCKER_VERSION needs to be repeated because
Expand All @@ -16,8 +16,7 @@ ARG MXNET_VERSION=1.8.0.post0
ARG PYSPARK_PACKAGE=pyspark==3.1.1
ARG SPARK_PACKAGE=spark-3.1.1/spark-3.1.1-bin-hadoop2.7.tgz

# Python 3.7 is supported by Ubuntu Bionic out of the box
ARG PYTHON_VERSION=3.7
ARG PYTHON_VERSION=3.8

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-cu"]
Expand Down