[Bug]

[ceteri]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

When our team runs Ray applications from command line, notebooks, etc., the code is running fine.
When we build and run containers, we get the following errors when we do not disable the Ray dashboard:
ImportError(DASHBOARD_DEPENDENCY_ERROR_MESSAGE)

This appears related to #19453 although they are reporting issues on Windows, and we're using Ubuntu Linux.

$ docker run --rm -p 8000:8000 -p 8265:8265 --shm-size=20gb -it ffurf:latest
ic| 'running on...', self.cluster_env: 'standalone'
ic| 'ffurf    ', __version__: '0.6'
Python implementation: CPython
Python version       : 3.8.5 
IPython version      : 7.29.0
 ImportError(DASHBOARD_DEPENDENCY_ERROR_MESSAGE)
Compiler    : GCC 7.3.0
OS          : Linux
Release     : 5.14.0-1005-oem
Machine     : x86_64
Processor   : x86_64
CPU cores   : 16
Architecture: 64bit

Hostname: 572a78f58f32

Git hash: f9d358f4bc21b678775ae94fec1b8008d66ed72c

Git repo: https://github.com/DerwenAI/ffurf.git

Git branch: env-var-k8s

Traceback (most recent call last):
  File "venv/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/opt/ffurf/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/main.py", line 425, in main
    run(app, **kwargs)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/main.py", line 447, in run
    server.run()
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/server.py", line 68, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/home/ray/anaconda3/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/server.py", line 76, in serve
    config.load()
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/config.py", line 448, in load
    self.loaded_app = import_from_string(self.app)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/opt/ffurf/venv/lib/python3.8/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/home/ray/anaconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./ffurf/asgi.py", line 11, in <module>
    front = Front(config="config.yml")
  File "./ffurf/src/front.py", line 72, in __init__
    ray.init(
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/worker.py", line 893, in init
    _global_node = ray.node.Node(
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/node.py", line 248, in __init__
    self.start_head_processes()
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/node.py", line 905, in start_head_processes
    self.start_dashboard(require_dashboard=True)
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/node.py", line 744, in start_dashboard
    self._webui_url, process_info = ray._private.services.start_dashboard(
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/_private/services.py", line 1260, in start_dashboard
    raise e from e
  File "/opt/ffurf/venv/lib/python3.8/site-packages/ray/_private/services.py", line 1187, in start_dashboard
    raise ImportError(DASHBOARD_DEPENDENCY_ERROR_MESSAGE)
ImportError: Not all Ray Dashboard dependencies were found. To use the dashboard please install Ray using `pip install ray[default]`.

Versions / Dependencies

Ray 1.7.1 install using pip:

ray[default] >= 1.7.1, < 1.8

running on base images:

• docker pull rayproject/ray:1.6.0-py38
• docker pull rayproject/ray:1.7.2-py38

Python implementation: CPython
Python version : 3.8.5
IPython version : 7.29.0

Compiler : GCC 7.3.0
OS : Linux
Release : 5.14.0-1005-oem
Machine : x86_64
Processor : x86_64
CPU cores : 16
Architecture: 64bit

Reproduction script

ray.init(
    ignore_reinit_error = True,
    include_dashboard = True,
    dashboard_host = "0.0.0.0",
    dashboard_port = 8265,
)

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[gerboland]

This helped me:

pip install aiohttp==3.7

[ceteri]

Thank you kindly @gerboland -
That helped fix the issue for us.

Also, we have:

• rolled back Ray to release 1.7.0 as suggested in [Bug]  #20073
• pinned the Arrow/Parquet C++ library to stable release 5.0.0-1 on Ubuntu (avoid Thrift mem leaks / seg faults in 6.0.0)

[ebr]

i belive this is closed by #20261