-
Notifications
You must be signed in to change notification settings - Fork 238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[hailtop.batch] eliminate unittest; use single, session-wide event loop #14097
Merged
danking
merged 39 commits into
hail-is:main
from
danking:cleanup-test-batch-service-backend-tests
Jan 9, 2024
Merged
[hailtop.batch] eliminate unittest; use single, session-wide event loop #14097
danking
merged 39 commits into
hail-is:main
from
danking:cleanup-test-batch-service-backend-tests
Jan 9, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danking
changed the title
cleanup
[hailtop.batch] eliminate unittest; use single, session-wide event loop
Jan 5, 2024
danking
force-pushed
the
cleanup-test-batch-service-backend-tests
branch
from
January 5, 2024 21:46
d0022ef
to
3ca8dd8
Compare
danking
force-pushed
the
cleanup-test-batch-service-backend-tests
branch
from
January 5, 2024 22:38
01fa04b
to
87b008c
Compare
danking
force-pushed
the
cleanup-test-batch-service-backend-tests
branch
from
January 5, 2024 22:47
87b008c
to
acbf966
Compare
@danking Looks like we need to upgrade |
…eate an http session outside of the event loop
danking
force-pushed
the
cleanup-test-batch-service-backend-tests
branch
from
January 8, 2024 19:39
99fe634
to
a36b2bf
Compare
@daniel-goldstein ok it finally works. |
daniel-goldstein
approved these changes
Jan 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #14130.
We pervasively assume:
Pytest (and newer version of IPython, afaict) violate this pretty liberally.
pytest_asyncio has explicit instructions on how to run every test in the same event loop. I've implemented those here.These instructions don't work. It seems that the reliable way to ensure we're using one event loop everywhere is to use pytest-asyncio < 0.23 and to define an event_loop fixture with scope'session'
.I also switched test_batch.py into pytest-only style. This allows me to use session-scoped fixtures so that they exist exactly once for the entire test suite execution.
Also:
RouterAsyncFS
methods must either be a static method or an async method. We must not create an FS in a sync method. Bothparse_url
andcopy_part_size
now both do not allocate an FS.httpx.py
now eagerly errors if the running event loop inrequest
differs from that at allocation time. Annoying but much better error message than this nonsense about timeout context managers.hail_event_loop
either gets the current thread's event loop (running or not, doesn't matter to us) or creates a fresh event loop and sets it as the current thread's event loop. The previous code didn't guarantee we'd get an event loop b/cget_event_loop
fails ifset_event_loop
was previously called.conftest.py
is inherited downward, so I lifted fixtures out of test_copy.py and friends and into a commonhailtop/conftest.py
make -C hail pytest-inter-cloud
for testing the inter cloud directory. You still need appropriate permissions and authn.FailureInjectingClientSession
creates anaiohttp.ClientSession
and therefore must be used while an event loop is running. Easiest fix was to make the test async.