Just trying to think through various cases where Machines belonging to MachineSets are deleted

1. MD is deleted

The following happens:

• MD goes away
• ownerRef triggers MS deletion
• MS goes away
• ownerRef triggers Machine deletion

=> Current PR doesn't help in this scenario, because the MS will already be gone when the deletionTimestamp is set on the Machines. In this case folks would have to modify the timestamps on each Machine individually.

I recently had a discussion with @vincepri, that we should maybe consider changing our MD deletion flow. Basically adding a finalizer on MD & MS, so MD & MS stick around until all Machines are gone. If we would do this, the MS => Machine propagation of the timeouts implemented here would help for this case as well

2. MD is scaled down to 0

The following happens:

• MD scales down MS to 0
• MS deletes Machine

=> This PR helps in this case because the timeouts are then propagated from MS to Machine

3. MD rollout

The following happens:

• Someone updates the MD (e.g. bump the Kubernetes version)
• MD creates a new MS and scales it up
• In parallel MD scales down the old MS to 0

=> In this scenario the current PR won't help, because the MD controller does not propagate the timeouts from MD to all MS (only to the new/current one, not to the old ones)

I see how this PR addresses scenario 2. Wondering if we want to solve this problem more holistically. (maybe I also missed some cases, not sure)

Here's what's going on... the use case is subtle, but an easy one to get trapped by.

• a MS is created with the default node drain timeout of (wait forever).
• The MS needs to scale down to zero (but not be deleted). The intent it to bring this MS back online at some point.
• The user discovers that the default node drain timeout is blocking the scale down to zero. The user likely only encounters this drain block the first time they scale down to zero because there are typically other nodes available during normal scale down operations which allows PDB to be satisfied.

The outcome is that the user is now trapped. They can't gracefully scale the MS down to zero because the default node drain timeout can't be updated on the machines. So the user is either forced to take some manual action to tear down the machines or delete the MS.

By allowing the node drain timeout to be modified while the machines are marked for deletion, we give the user a path to unblock themselves using the top level api (either MS or MD) rather than mutating individual machines or performing some other manual operation.

Yup got it, and makes sense. I was just saying that there are various cases with MD+MS where a MS is scaled down to zero. The implementation only coverd one of them. But it's fine for me to consider addressing the others in separate PRs. Would be probably just good to open an issue so we can track that (I can do that)