docker image created (using instruction from idmtools_platform_comps/ssmt_image) may hit error "OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 48: Operation not permitted"

[tinghf]

Table of Contents

- [Description](#description)
- [What I Did](#what-i-did)

• Extras

Description

docker image (using instruction from idmtools_platform_comps/ssmt_image) created may hit error "OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 48: Operation not permitted" when running in COMPS docker worker.

See this COMPS ticket for details:
https://idmod.visualstudio.com/COMPS/_workitems/edit/4612

What I Did

Kurt hit an issue while running docker worker on some of the docker container he built.
it's failing from on some numpy.core.multiarray import errors, like following:

OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 48: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 46 of 48: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 47 of 48: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
Traceback (most recent call last):
File "/mnt/idm2/home/kfrey/workitems/0b6/725/696/0b672569-67f3-ed11-aa06-b88303911bc1/Assets/emod_reduce.py", line 13, in
from idmtools_platform_comps.ssmt_work_items.comps_workitems import SSMTWorkItem
File "/usr/local/lib/python3.9/site-packages/idmtools_platform_comps/init.py", line 8, in
from idmtools_platform_comps.plugin_info import COMPSPlatformSpecification
File "/usr/local/lib/python3.9/site-packages/idmtools_platform_comps/plugin_info.py", line 10, in
from idmtools_platform_comps.comps_platform import COMPSPlatform
File "/usr/local/lib/python3.9/site-packages/idmtools_platform_comps/comps_platform.py", line 12, in
from idmtools.entities.experiment import Experiment
File "/usr/local/lib/python3.9/site-packages/idmtools/entities/experiment.py", line 17, in
from idmtools.builders import SimulationBuilder
File "/usr/local/lib/python3.9/site-packages/idmtools/builders/init.py", line 7, in
from idmtools.builders.simulation_builder import SimulationBuilder
File "/usr/local/lib/python3.9/site-packages/idmtools/builders/simulation_builder.py", line 6, in
import pandas as pd
File "/usr/local/lib64/python3.9/site-packages/pandas/init.py", line 11, in
import(_dependency)
File "/usr/local/lib64/python3.9/site-packages/numpy/init.py", line 141, in
from . import core
File "/usr/local/lib64/python3.9/site-packages/numpy/core/init.py", line 23, in
from . import multiarray
File "/usr/local/lib64/python3.9/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/usr/local/lib64/python3.9/site-packages/numpy/core/overrides.py", line 6, in
from numpy.core._multiarray_umath import (
KeyboardInterrupt

the couple of threads that Kurt located all seems to have a solution to fix this by updating to a later version of docker:

https://medium.com/nttlabs/ubuntu-21-10-and-fedora-35-do-not-work-on-docker-20-10-9-1cd439d9921

cvat-ai/cvat#6038

Also things seems to work when Kurt built another container using a older Ubuntu (20.04):

https://comps.idmod.org/#explore/WorkItems?filters=Id=c4f40db7-79f3-ed11-aa06-b88303911bc1

Extras

[devclinton]

@shchen-idmod @tinghf @ZDu-IDM - We are looking at adding this issue to our next maintenance release. I am trying to understand the resolution you provided @tinghf . It appears updating docker works, from the link you shared, but that would be a comps update? Was there something to update in our container directly?