Ensure client key cancellation uses ordered messages

[fjetter]

Another breakout from #7564

I can't specifically recall where I ran into this but while working on the refactor I somehow ran into race conditions around cancellation that prompted me to rewrite this section.

---

All administrative message, e.g. compute, release, cancel are sensitive to ordering. The current cancellation mechanism works around this by "waiting" for a task to show up because the current cancel was using an unordered RPC call (see also #7480)
The only other reason why one would want to poll at this point is if another client could cancel a given key. There is no API for this. User exposed APIs require the existence of a future object which, given message ordering, ensure that the task exists on the scheduler.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       26 files  +       2         26 suites  +2   5d 14h 51m 55s ⏱️ + 5d 4h 6m 33s
  3 491 tests +     79    3 387 ✔️ +     80     103 💤 ±    0  1 ❌  - 1 
44 123 runs  +3 966  42 055 ✔️ +3 740  2 066 💤 +227  2 ❌  - 1 

For more details on these failures, see this check.

Results for commit 83695f1. ± Comparison against base commit f9306da.

♻️ This comment has been updated with latest results.

[hendrikmakait]

Thanks, @fjetter!