Setup time for parallelize worker testing is excessive for singe test case runs

[dhh]

When running tests with parallelized workers, we seem to be doing the full database setup/schema load/teardown/fixtures loading even when only running a single test suite rather than all the suites.

On a 10-core machine, that means doing that 10 times (one database per core), which takes a long time, especially when you're just trying to iterate over a single test case. When parallelize workers is turned off, there's no setup/teardown being done. Just resetting the fixtures.

For my 10-core iMac Pro, running a 2-test test suite takes 0.4s without parallelize workers and 6s with parallelize workers.

When I run the entire suite, it's clearly better to run with parallelize workers (13s vs 38s!), but when running a single suite, it's a pain.

It seems like we should differentiate between "./bin/rails test", which should run with full parallelization, and "./bin/rails test tests/model/case.rb", which should not.

[dhh]

cc @jhawthorn

[dhh]

Thinking this might be as simple as avoiding the parallelization path when the test runner is being asked to run just one case. You might still be paying the high tax early on in your app development on a many core machine, but you could always tweak the default settings in test_helper.rb while that's the case.

[ghost]

What happened to Convention over configuration?

[dhh]

It's exactly the refinement of the convention that this issue is addressing: run parallelized workers with setup cost when the pay off is worth it, don't when it's not, and let the user decide when it's ambiguous (tiny app / many cores).

[ghost]

Wouldn't the sensible convention for a newly generated app be to skip parallelization in either case?

[dhh]

No, because the setup cost with a tiny schema is likely to be tiny too. And that would then require changing configuration once the tip-over point is reached, which is likely to be forgotten. The hurt here is specifically with large app / many cores / just running a few test cases in an iterative loop.

[ghost]

Okay, seems logical, since it's difficult to compute the tip-over point, it would have to be remembered by a human to change it. Sorry for interrupting.

[jhawthorn]

I've also run into slow setup on machines with many cores. I want to investigate only re-initializing the test databases when the schema changes (which hopefully makes parallelization setup fast enough to be a benefit even for a single suite). My initial thinking is to store something to track this in ar_internal_metadata. I'll try to test the idea this week.

[dhh]

That would be even better, @jhawthorn ❤️

[alexpooley]

We can lazily setup each database instead of having it ready up front. In practice this means you only run the after fork calls after popping the first job from the queue.

Partial solution at PR #36809

I was unable to test on 10 cores. Donations welcome 😄

[jhawthorn]

This should hopefully be much better now with #36873 (also backported to 6-0-stable in 4f912de). The first run of the test suite in parallel (and each first after a schema change) will create the databases and load the schema, successive runs will reuse the previously created databases.

[dhh]

Wonderful!

…

On Fri, Aug 9, 2019 at 4:26 PM John Hawthorn ***@***.***> wrote: This should hopefully be much better now with #36873 <#36873> (also backported to 6-0-stable). The first run of the test suite in parallel (and each first after a schema change) will create the databases and load the schema, successive runs will reuse the previously created databases. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36807?email_source=notifications&email_token=AAAAVNICFEO6MDCSTUCRQ3TQDX4JTA5CNFSM4IHWYZQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4AACTQ#issuecomment-520094030>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAVNLOJ6EWBIDPG2Z7VGLQDX4JTANCNFSM4IHWYZQA> .