[BUG] CI Broken with the latest distributed release 2021.10.0

[VibhuJawa]

[BUG] CI Broken with distributed 2021.10.0

Minimal Complete Verifiable Example:

Create Environment

conda env create -f ci/environment-latest.yaml --name=dask-ml-dev

Run Tests:

pytest .

=================================================================================================================== ERRORS ===================================================================================================================
__________________________________________________________________________________________ ERROR collecting tests/model_selection/test_hyperband.py __________________________________________________________________________________________
tests/model_selection/test_hyperband.py:163: in <module>
    def test_hyperband_patience(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
_________________________________________________________________________________________ ERROR collecting tests/model_selection/test_incremental.py _________________________________________________________________________________________
tests/model_selection/test_incremental.py:338: in <module>
    def test_search_plateau_patience(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
______________________________________________________________________________________ ERROR collecting tests/model_selection/test_incremental_warns.py ______________________________________________________________________________________
tests/model_selection/test_incremental_warns.py:11: in <module>
    def test_warns_decay_rate(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
____________________________________________________________________________________________ ERROR collecting tests/model_selection/test_keras.py ____________________________________________________________________________________________
tests/model_selection/test_keras.py:47: in <module>
    def test_keras(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
___________________________________________________________________________________________ ERROR collecting tests/model_selection/test_pytorch.py ___________________________________________________________________________________________
tests/model_selection/test_pytorch.py:29: in <module>
    def test_pytorch(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
_____________________________________________________________________________________ ERROR collecting tests/model_selection/test_successive_halving.py ______________________________________________________________________________________
tests/model_selection/test_successive_halving.py:15: in <module>
    def test_basic_successive_halving(c, s, a, b):
/datasets/vjawa/miniconda3/envs/dask-ml-dev-oct-16/lib/python3.8/site-packages/distributed/utils_test.py:922: in _
    raise RuntimeError("gen_cluster only works for coroutine functions.")
E   RuntimeError: gen_cluster only works for coroutine functions.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================================= 6 errors in 1.04s ==============================================================================================================

What happened:

The above pytests seem to be failing with the latest distributed release. Above works with distributed/dask=2021.9.1 .

Environment:

• Dask version: 2021.10.0
• Python version: 3.8.12
• Operating System: linux
• Install method (conda, pip, source): conda

[jakirkham]

Looks related to PR ( dask/distributed#5446 )

[jakirkham]

cc @fjetter (for awareness)

[jrbourbeau]

This also popped up in dask/dask#8298. Specifically these changes

[VibhuJawa]

Thanks for the help and the context guys, Working on a push on this today.

[fjetter]

FWIW, most calls can be easily fixed by defining the functions as async. I was facing this problem since I was scheduling the test runtime as a asyncio.Task and that didn't fit well with the tornado gen.coroutine thing we've had in there before. However, I wasn't aware that this was used elsewhere (obviously it is 😅 ) and I think I can fix that, if need be. I'm wondering if that already warrants a patch release? 🤔

[jrbourbeau]

I wasn't aware that this was used elsewhere (obviously it is 😅 )

Sorry this one slipped past me

I'm wondering if that already warrants a patch release? 🤔

I'm comfortable not issuing a patch release for now. That said, if we're able to include some backwards compatibility code in the next release that would be great. Maybe we wait to see if other raise similar issues? If so, then we could reevaluate a patch release