Dask Kubernetes v2 (Stability) Release

[jacobtomlinson]

Dask Kubernetes Summer Roadmap

Note
Creating in rapidsai/deployment so I can use tasklists. When tasklists are GA I'll migrate this issue to the dask/dask-kubernetes repo.

At the end of the summer I want to release V2 of the Dask Kubernetes Operator and fully remove the deprecated classic implementations. This issue outlines the roadmap that we need to complete to get us to a point where we can do that.

High-level goals:

• Improve stability
• Ensure feature completeness compared to other implementations

Some of the sections here may want to be split off into separate issues, and some tasks may want to be broken down into smaller chunks. But this will be the high-level milestone tracker issue for this work.

Features

Cluster idle timeout

Cleaning up idle clusters automatically becomes critical for cost-reduction when deploying at scale. Especially when using GPUs.

Tasks

Beta Give feedback

1. Add idle cluster cleanup dask/dask-kubernetes#667
   enhancement help wanted operator +3
   

Full Istio support

Currently we have partial Istio support where the scheduler uses it but workers do not. This can be a blocker for clusters that enforce Istio on all comms.

Tasks

Beta Give feedback

1. Support Istio and using the envoy proxy dask/dask-kubernetes#497
   operator +1

UX Improvements

UX can always be improved

Tasks

Beta Give feedback

1. Add more additional printer columns to resources dask/dask-kubernetes#699
   enhancement operator +2

Fixes

Replace Pod resources with higher abstractions like Deployment or at least ReplicaSet

Currently we manage bare Pods. There are downsides to this such as pods not being recreated when they are evicted from a node. It would be good to explore higher-level resources and how they could be used to simplify our controller logic.

Tasks

Beta Give feedback

1. Recreate scheduler if lost dask/dask-kubernetes#606
   enhancement operator +2
   
2. Worker group doesn't recover after pod deletion dask/dask-kubernetes#603
   bug operator +2
   
3. Add backoffLimit to DaskJobs dask/dask-kubernetes#695
   enhancement operator +2

Ensure patches to DaskCluster and DaskWorkerGroup are propagated to child resources

In the context of CRUD we only have create, read and delete implemented for our resource. We also need to correctly handle updating them.

Tasks

Beta Give feedback

1. k8s Operator: DaskCluster spec updates do not propagate to pods dask/dask-kubernetes#636
   enhancement operator +2
   

Ensure scaling/autoscaling is solid

Some users are reporting unwanted behaviour when autoscaling at scale. This needs to be solid.

Tasks

Beta Give feedback

1. Improper scaling of dask workers dask/dask-kubernetes#677
   bug operator +2
2. Issue with AdaptiveScaling using Dask Kubernetes Operator dask/dask-kubernetes#674
   bug needs info operator +3
3. Autoscaler closes workers as others are starting/scaling up dask/dask-kubernetes#659
   bug operator +2
4. Adapt method difference between classic and operator KubeCluster dask/dask-kubernetes#645
   bug needs info operator +3
5. Add batched worker provisioning to Dask Cluster spawning on Kubernetes dask/dask-kubernetes#733
   enhancement operator +2
   

Input sanitisation

Currently we rely on bad configuration being validated by the Kubernetes API, but this doesn't always happen as we expect. We should do more checking and sanitization before calling the Kubernetes API.

Tasks

Beta Give feedback

1. Typos in resources silenty get ignored dask/dask-kubernetes#650
   bug operator +2
   
2. Invalid KubeCluster kwargs raises confusing exception dask/dask-kubernetes#698
   bug operator +2
   
3. KubeCluster Input sanitising dask/dask-kubernetes#682
   bug operator +2
4. Operator fails if nodeport out of allowed range dask/dask-kubernetes#536
   bug operator +2

Controller idempotency

The controller event handlers should be idempotent and should be able to be called multiple times. Today they are not which can cause problems when restarting the controller while operations are running.

Tasks

Beta Give feedback

1. Cluster creation fails with scheduler AlreadyExists error dask/dask-kubernetes#654
   bug operator +2
   

Hygeine/Tech Debt

Migrate Kubernetes client library to kr8s

Today we use pykube-ng, dask_kubernetes.aiopykube, kubernetes_asyncio and subprocess/kubectl to interact with the Kubernetes API. We should consolidate everything around kr8s which was spun out from here with the intention of unifying our API usage.

Tasks

Beta Give feedback

1. Consolidate dask-kubernetes client usage around kr8s #220
   18 of 18
   +0
   

Other

Tasks

Beta Give feedback

1. Retire Classic KubeCluster #316
   +0
   
2. Create new helm chart to replace dask/dask with one that uses a DaskCluster instead of scheduler and worker Deployment/Service resources
3. Retire HelmCluster #317
   +0
   

[tasansal]

@jacobtomlinson would it make sense to add this to the list as well?

dask/dask-kubernetes#605

[jacobtomlinson]

@tasansal We have #603 on the list, which that PR closes 😊

[jacobtomlinson]

I'm going to close this epic out as done. Not all tasks here are complete, but they will be prioritised as part of future work.