[Bug] Specifying runtime_env leads to failure on Apple ARM64

[joseph-long]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

Having recently obtained an Apple ARM laptop, I'm in the process of bringing up my development environment on it. I've followed https://docs.ray.io/en/master/installation.html#m1-mac-apple-silicon-support to set up, and verified that ray start --head and ray microbenchmark work, but my pipeline does not.

I've narrowed it down to the runtime_env argument to ray.init(), which leads to this message:

% python foo.py
Traceback (most recent call last):
  File "/Users/josephlong/devel/test/ray_bug/foo.py", line 14, in <module>
    res = ray.get(ref)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/worker.py", line 1627, in get
    raise value
ray.exceptions.RuntimeEnvSetupError: The runtime environment for this task or actor failed to be installed. Corresponding error logs should have been streamed to the driver's STDOUT.
(raylet) [2021-11-21 12:20:27,923 E 56546 3176166] agent_manager.cc:134: Not all required Ray dependencies for the runtime_env feature were found. To install the required dependencies, please run `pip install 'ray[default]'`.

Versions / Dependencies

ray 1.8
Python 3.9
macOS 12 Monterey

Reproduction script

import ray

ray_init_kwargs = {'runtime_env': {'env_vars': {'RAY_USER_SETUP_FUNCTION': 'logging.basicConfig'}}}
# ray_init_kwargs = {}  # comment above and uncomment this and it'll work

@ray.remote
def bar(a):
    print(a)
    return a

if __name__ == "__main__":
    ray.init(**ray_init_kwargs)
    ref = bar.remote(1)
    res = ray.get(ref)
    print(res)

Anything else

I had a look at the code that produces that error and was unable to figure out which dependency it needed. It may be that this is just a dependency issue...

[rkooo567]

Cc @edoakes

[jiaodong]

cc: @architkulkarni can you help to provide more context if runtime_env is supposed to work and supported for Apple ARM64 ?

[shrekris-anyscale]

I believe the runtime_env feature requires the dependencies from ray[default]. @joseph-long Could you try running pip install "ray[default]" and then retrying the reproduction script? See the Note at the top of the "Runtime Environments" section of this doc for more info.

[joseph-long]

That is how I installed it initially, but for completeness:

% pip install 'ray[default]'                                                                                                                                                                                                                                                                                                                           (split_vapp_trap !)
Requirement already satisfied: ray[default] in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (1.8.0)
Requirement already satisfied: attrs in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (20.3.0)
Requirement already satisfied: filelock in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (3.4.0)
Requirement already satisfied: grpcio>=1.28.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (1.42.0)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (1.0.2)
Requirement already satisfied: numpy>=1.19.3 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (1.21.4)
Requirement already satisfied: protobuf>=3.15.3 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (3.19.1)
Requirement already satisfied: click>=7.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (8.0.3)
Requirement already satisfied: jsonschema in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (4.2.1)
Requirement already satisfied: pyyaml in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (6.0)
Requirement already satisfied: redis>=3.5.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (4.0.1)
Requirement already satisfied: requests in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (2.26.0)
Requirement already satisfied: gpustat>=1.0.0b1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (1.0.0b1)
Requirement already satisfied: colorful in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (0.5.4)
Requirement already satisfied: prometheus-client>=0.7.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (0.12.0)
Requirement already satisfied: opencensus in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (0.8.0)
Requirement already satisfied: aiohttp>=3.7 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (3.8.1)
Requirement already satisfied: aiohttp-cors in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (0.7.0)
Requirement already satisfied: aioredis<2 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (1.3.1)
Requirement already satisfied: py-spy>=0.2.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from ray[default]) (0.3.11)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (1.2.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (4.0.1)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (1.2.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (2.0.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (1.7.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aiohttp>=3.7->ray[default]) (5.2.0)
Requirement already satisfied: hiredis in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from aioredis<2->ray[default]) (2.0.0)
Requirement already satisfied: six>=1.7 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from gpustat>=1.0.0b1->ray[default]) (1.16.0)
Requirement already satisfied: psutil in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from gpustat>=1.0.0b1->ray[default]) (5.8.0)
Requirement already satisfied: blessed>=1.17.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from gpustat>=1.0.0b1->ray[default]) (1.19.0)
Requirement already satisfied: nvidia-ml-py3>=7.352.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from gpustat>=1.0.0b1->ray[default]) (7.352.0)
Requirement already satisfied: deprecated in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from redis>=3.5.0->ray[default]) (1.2.13)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from jsonschema->ray[default]) (0.18.0)
Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from opencensus->ray[default]) (2.2.2)
Requirement already satisfied: opencensus-context==0.1.2 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from opencensus->ray[default]) (0.1.2)
Requirement already satisfied: idna<4,>=2.5 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from requests->ray[default]) (3.1)
Requirement already satisfied: certifi>=2017.4.17 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from requests->ray[default]) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from requests->ray[default]) (1.26.7)
Requirement already satisfied: typing-extensions>=3.6.5 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from async-timeout<5.0,>=4.0.0a3->aiohttp>=3.7->ray[default]) (4.0.0)
Requirement already satisfied: wcwidth>=0.1.4 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from blessed>=1.17.1->gpustat>=1.0.0b1->ray[default]) (0.2.5)
Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (2.3.3)
Requirement already satisfied: setuptools>=40.3.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (59.1.1)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (1.53.0)
Requirement already satisfied: wrapt<2,>=1.10 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from deprecated->redis>=3.5.0->ray[default]) (1.13.3)
Requirement already satisfied: rsa<5,>=3.1.4 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (4.2.4)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]) (0.4.8)

[joseph-long]

% RAY_BACKEND_LOG_LEVEL=debug python foo.py
[2021-11-22 17:11:38,014 I 94310 3885808] logging.cc:190: Set ray log level from environment variable RAY_BACKEND_LOG_LEVEL to -1
Traceback (most recent call last):
  File "/Users/josephlong/devel/test/ray_bug/foo.py", line 14, in <module>
    res = ray.get(ref)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/worker.py", line 1627, in get
    raise value
ray.exceptions.RuntimeEnvSetupError: The runtime environment for this task or actor failed to be installed. Corresponding error logs should have been streamed to the driver's STDOUT.
(raylet) [2021-11-22 17:11:38,150 E 94316 3885842] agent_manager.cc:134: Not all required Ray dependencies for the runtime_env feature were found. To install the required dependencies, please run `pip install 'ray[default]'`.

[edoakes]

@joseph-long could you please try running import ray.dashboard.optional_deps in a python shell and provide the output? That is the underlying check happening here to determine if the dependencies are available.

[joseph-long]

Thanks, that's the diagnostic I was missing! Not an ARM64 thing at all, it turns out. It's just #19940 again.

>>> import ray.dashboard.optional_deps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/dashboard/optional_deps.py", line 8, in <module>
    import aiohttp.signals
ModuleNotFoundError: No module named 'aiohttp.signals'

Fixed with pip install 'aiohttp<3.8'

(Would be great if that ImportError output were part of the Corresponding error logs should have been streamed to the driver's STDOUT. 😃)