-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Runtimes slow down dramatically proportionally with feature file size #331
Comments
Actually, you shouldn't. The behavior you observe, that splitting source files leads to a significant performance gain, seems to be buggy, weird and unexpected. There shouldn't be any significant difference. Should be investigated and fixed. The idea we have in mind for now is that a |
@ilslv do you have any suggestions on this? |
@tgsmith61591 can you share a little bit more about the characteristics of the testing suite? Is it async or sync heavy, or maybe appropriately both? Can you share the |
Hey @ilslv and @tyranron, the test suite is very tl;dr
|
Thank you for the reproduction repo! I'll definitely take a look hopefully this weekend. |
Hey @ilslv, any chance you got a chance to look at this? |
not yet, unfortunately 😢 |
Any update on this issue @ilslv ? |
We maintain a large collection of feature files that feed nightly regression tests, the runtime of which has grown significantly recently. These are generally maintained in logically separated feature files, and leverage
Scenario Outline
tables, sometimes with 100-200 scenarios per feature file.In experimenting with optimizations on a single tag, I tried splitting a feature file into 4 and observed a massive performance gain. Here is my baseline:
Here are the same exact tests divided over 4 feature files:
For reference, here is how we're running (note that
local_test_jobs
keeps Bazel from trying to do its own parallelism, instead delegating the concurrency to the cucumber engine):$ bazel run //my-crate:my-test-suite \ --local_test_jobs=1 -- \ --concurrency 16 \ --tags="@my-cool-tag"
Several questions I have after observing this major performance difference:
scenario
level, but now I'm wondering whether it's actually at the feature levelconcurrency
to the number of feature files?cucumber-rs
engine?The text was updated successfully, but these errors were encountered: