KubeCluster is shut down automatically even if shutdown_on_close is False

[Paikan]

Describe the issue:

I am trying to use metaflow & dask on k8s. I am using the k8s dask kubernetes operator. I would like to have an initial step in metaflow that creates a dask cluster that will be used by all next steps with the following code:

cluster = KubeCluster(
                name=self.dask_cluster_name,
                n_workers=2,
                resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "2Gi"}},
                env={"EXTRA_PIP_PACKAGES": "s3fs==2023.12.2"},
                shutdown_on_close=False,
            )

# Same behavior for
cluster = KubeCluster(
                name=self.dask_cluster_name,
                n_workers=2,
                resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "2Gi"}},
                env={"EXTRA_PIP_PACKAGES": "s3fs==2023.12.2"},
            )
cluster.shutdown_on_close = False

it works as intended but as soon as the pod that ran the start step and created the dask cluster is over the cluster is closed. I was expecting shutdown_on_close set to False would prevent this behavior.

Environment:

• Dask version: 2024.1.1
• Python version: 3.10.6
• Operating System: Ubuntu
• Install method (conda, pip, source): pip

[jacobtomlinson]

Sorry that this issue got missed! I am also surprised by this behaviour. Are you still able to reproduce this on recent versions of dask-kubernetes?