Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import tensorflow failing on ml-notebook #536

Open
Timh37 opened this issue Apr 25, 2024 · 0 comments
Open

Import tensorflow failing on ml-notebook #536

Timh37 opened this issue Apr 25, 2024 · 0 comments

Comments

@Timh37
Copy link

Timh37 commented Apr 25, 2024

Describe the bug
Using the most recent image of the Tensorflow GPU ML-notebook, import tensorflow leads to the following error:

2024-04-23 07:46:16.210178: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-23 07:46:16.210210: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-23 07:46:16.210216: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-04-23 07:46:16.217822: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

and as a result, model training fails.

To Reproduce
Start up a server with the ml-notebook image and runimport tensorflow.

Expected behavior
To import tensorflow without issues.

Docker Image Version (e.g. quay.io/pangeo/ml-notebook:2023.02.27):
NVIDIA Tesla T4, 24GB RAM, 8 CPUs, Pangeo Tensorflow ML Notebook

Infrastructure (Where you are running this image):
The 2i2c JupyterHub for LEAP.

Additional context
If I use an older version of the image (quay.io/pangeo/ml-notebook:2023.05.08) the errors disappear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant