Reduced benchmarks #2166
Reduced benchmarks #2166
Conversation
…rations) added a check on "#skip-benchmarks" in the pr description
…rations) added a check on "#skip-benchmarks" in the pr description
…rations) added a check on "#skip-benchmarks" in the pr description
…rations) added a check on "#skip-benchmarks" in the pr description
…rations) added a check on "#skip-benchmarks" in the pr description
Codecov Report
@@ Coverage Diff @@
## main #2166 +/- ##
=========================================
Coverage 80.91% 80.91%
Complexity 3287 3287
=========================================
Files 233 233
Lines 12043 12043
Branches 1594 1594
=========================================
Hits 9745 9745
Misses 1715 1715
Partials 583 583 Continue to review full report at Codecov.
|
on pr only one device will be tested nightly all devices will be tested
| @@ -1,9 +1,18 @@ | |||
| name: "Integration Tests - Benchmarks" | |||
| on: | |||
| schedule: | |||
There was a problem hiding this comment.
Just thinking if it's necessary, because they will anyway run on every merge/push to main? If there were no changes on main and a cron job kicks in, it's still useless probably, because there were nothing new on main?
There was a problem hiding this comment.
you may be right on this
let's remove pushing to main: we may have multiple pushes to main daily, so we could end up with a lot of tests running on saucelabs, risking timeouts and slowdowns. Also, we would still run the single benchmark for every pr, so running only nightly is enough at the moment. We can still change it later
There was a problem hiding this comment.
Yeah, sounds good 👍 just need some kind of reminder on slack or somewhere to check the benchmarks from the nightly builds
There was a problem hiding this comment.
We don't need to do it now, but would be cool to either post on slack or automatically create an issue if there were regressions after the nightly build
There was a problem hiding this comment.
actually I guess it's pretty straightforward - if a test run failed, we just use some action to create an issue and that's it :D most important is that there shouldn't be flaky tests to not overwhelm us with issues haha
There was a problem hiding this comment.
it shouldn't be the case, even because when a test times out it succeeds 😬
|
Just one small comment, otherwise LGTM 🚀 You already got my approval anyway |
📜 Description
Reduced benchmarks to only run on android/java packages (not on integrations)
On every pull request the benchmarks will be run on one device only.
The full benchmark device suite will be tested nightly.
#skip-changelog
💡 Motivation and Context
The benchmarks currently take ~1 hour to run on saucelabs, more if a device times out. The situation is going to get worse as more tests are added.
To alleviate the situation we run the tests only on one device, so that this workflow should take ~10-15 minutes (plus build time). Also, running benchmarks on just one device reduces the risk of timeouts from saucelabs.
💚 How did you test it?
📝 Checklist
🔮 Next steps
We will monitor the situation and will make further changes in the future if needed, but at the moment these changes should be enough