New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bolster performance of test suite execution #2874
Comments
Quite many tests run the server on some hardcoded port, so a bit of care might be needed to enable parallel tests. |
this makes sense, sounds like interesting problem to overcome, perhaps we can ask for the available port from os 🤔 |
An excellent goal, but that is going to be a painful task. As mentioned, because we're doing a lot of io, there are a lot of tests that are much more integration than unit. There's a lot of baked in synchronization and state management. It will be a huge effort to overcome and im not sure we'd be able to cover all of the same use cases. Port selection alone isn't the only obstacle. Perhaps a more realistic goal may be to separate the test suites into a few subgroups and run each of those simultaneously. But not sure that'd be much better as it would explode the number of runners. |
Installing dependencies is also a big slowdown with the current CI. Perhaps that can be sped up somehow? Using a Docker image with everything preinstalled, for instance? |
That's a great idea. I'm all for this. We did this at work and it was a huge time saver. |
@ahopkins @Tronic Perhaps Can you please share some links with examples ? |
@iAndriy This is correct, the vast majority of time is consumed in pytest. Different platforms are run in parallel but it still takes long, and this is cumbersome when running them locally, too. Many tests use But going through them to optimize and improve, making sure not to miss anything important, is a huge task that I doubt there is manpower to. Test parallelization might yield some easy results but it still needs care not to make the tests fail at random. Also looking at the few particularly slow tests in detail might yield useful speedup. As would reducing the number of combinations in parametrized tests. And reducing sleep/timeout durations (as long as no random failures occur then). |
@Tronic Thanks for elaborating, a lot of useful details I haven't know about. yeah, it's important to keep tests deterministic, flakiness is a huge problem and being conscious of it during optimisation is important. As per your earlier suggestion re-organizing tests and adding dynamic ports to enable parallelism shall do the work. Surprisingly, some of the tests fail on my local machine but work during CI/CD, so maybe adding some utilities to run tests in docker (on local machine) would be useful too. |
I found quite many tests failing both on local MacOS and some versions of Linux. These were addressed in #2844 in case you want to dig up the diff or specific commits from that branch. With those changes I got the tests passing on MacOS and various Linux distros, not sure if I tried Windows though. The current main still fails many tests on my Macbook. |
Is there an existing issue for this?
Is your feature request related to a problem? Please describe.
As software engineer I would like to have fast test execution cycle, so the jobs associated with CI pipeline complete faster, hence reducing resources usage time.
Current execution of tests takes 7+ minutes (though it depends on retries), ideally I would like it to take few minutes and going under ten minutes would be a good start.
Describe the solution you'd like
I would like to use pytest-xdist to parallelize test execution.
Additional context
https://pypi.org/project/pytest-xdist/
The text was updated successfully, but these errors were encountered: