Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paused workers shouldn't steal tasks (flaky test_avoid_paused_workers) #5664

Closed
crusaderky opened this issue Jan 17, 2022 · 0 comments · Fixed by #5665
Closed

Paused workers shouldn't steal tasks (flaky test_avoid_paused_workers) #5664

crusaderky opened this issue Jan 17, 2022 · 0 comments · Fixed by #5665
Assignees
Labels
flaky test Intermittent failures on CI.

Comments

@crusaderky
Copy link
Collaborator

crusaderky commented Jan 17, 2022

#5431 changed Scheduler.decide_worker to stop it from assigning new tasks to workers with paused or closing_gracefully status.
However, those workers are still stealing tasks - effectively negating the benefits of the PR.

This is reflected by the flakiness of test_avoid_paused_workers; the test frequently hangs in CI on the lines

    while (len(w1.tasks), len(w2.tasks), len(w3.tasks)) != (4, 0, 4):
        await asyncio.sleep(0.01)

Above, w2 is paused. However, the tuple ends up looking like (3, 1, 4) instead. If you add await wait(futures), the test will start hanging deterministically since a task is always stolen from one of the running workers to the paused one, and there it sits forever since nothing steals it back.

Adding , config={"distributed.scheduler.work-stealing": False} to the gen_cluster decorator makes the issue disappear.

@crusaderky crusaderky self-assigned this Jan 17, 2022
@crusaderky crusaderky added the flaky test Intermittent failures on CI. label Jan 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky test Intermittent failures on CI.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant