You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a Prefect agent; would like to use Dask in K8S. We don't use Helm, so having it self-contained in the Prefect flow would be neat. How do I make this happen?
This code doesn't work when called from my laptop:
$ python deploy-dask.py
Creating scheduler pod on cluster. This may take some time.
Traceback (most recent call last):
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/comm/core.py", line 318, in connect
handshake = await asyncio.wait_for(comm.read(), time_left())
File "/Users/h/.pyenv/versions/3.8.12/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/deploy/spec.py", line 318, in _start
await super()._start()
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 78, in _start
comm = await self.scheduler_comm.live_comm()
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/core.py", line 753, in live_comm
comm = await connect(
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/comm/core.py", line 323, in connect
raise OSError(
OSError: Timed out during handshake while connecting to tcp://dask-cluster.flows:8786 after 30 s
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "deploy-dask.py", line 2, in <module>
cluster = KubeCluster(
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/dask_kubernetes/core.py", line 466, in __init__
super().__init__(**self.kwargs)
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/deploy/spec.py", line 283, in __init__
self.sync(self._start)
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 214, in sync
return sync(self.loop, func, *args, **kwargs)
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
raise exc.with_traceback(tb)
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
result[0] = yield future
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/dask_kubernetes/core.py", line 595, in _start
await super()._start()
File "/Users/h/dev/haf/data-pipelines/.venv/lib/python3.8/site-packages/distributed/deploy/spec.py", line 322, in _start
raise RuntimeError(f"Cluster failed to start. {str(e)}") from e
RuntimeError: Cluster failed to start. Timed out during handshake while connecting to tcp://dask-cluster.flows:8786 after 30 s
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1193a8f40>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1193bd070>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x11942b3a0>, 80.372319483)]']
connector: <aiohttp.connector.TCPConnector object at 0x1193a8ca0>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x1193d01c0>, 80.299328274)]']
Is it just taking too long to install packages? In any case; how do I configure Dask with Prefect, ensuring we can reconnect to the cluster if need-be and that the cluster auto-scaled away workers on a by-need basis?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi,
I have a Prefect agent; would like to use Dask in K8S. We don't use Helm, so having it self-contained in the Prefect flow would be neat. How do I make this happen?
Here's what I've tried
I have a bit of code:
This code doesn't work when called from my laptop:
on the pod:
Is it just taking too long to install packages? In any case; how do I configure Dask with Prefect, ensuring we can reconnect to the cluster if need-be and that the cluster auto-scaled away workers on a by-need basis?
Context, the worker-spec.yaml:
Beta Was this translation helpful? Give feedback.
All reactions