Add support for Rails 6 built-in Parallel Testing

[zoras]

Use Rails 6 built-in parallelizer API for running parallel RSpec executors

https://edgeguides.rubyonrails.org/testing.html#parallel-testing

rails/rails#31900 (comment)

[JonRowe]

We'd love to support parallel tests with RSpec itself, its a rather large task however. If we get parallelisation support into rspec-core you can be sure we'll integrate with the rails helpers.

[jrochkind]

Note that linked #1254 says:

There are bunch of gems that allow for parallel execution, but all of these are process based and not thread based.

I believe the parallel testing now built into Rails can be process-based or thread-based, and by default is process-based.

Is supporting Rails process-based parallel testing less of a challenge? It looks like one of the main things Rails does for you when supporting this is setting up a separate database for each test worker, as well as orchestrate getting each worker to run different tests and aggregating the results.

[JonRowe]

In the short term all we could offer is to help integrate 3rd party parallel testing with Rails tools.

Our own parallel journey requires changes to rspec-core which are mostly identical for process or thread based.

Please note there are parallel testing extensions for RSpec already!

[p8]

Discourse now has a solution based on parallel_tests discourse/discourse#7778

[iainbeeston]

Is there any update on supporting the activesupport parallelize helper in rspec-rails, now that rails 6 has been released?

[iainbeeston]

Sorry, I meant to include a link with more details:

rails/rails#31900

[JonRowe]

No, RSpec itself needs to support parallelisation in order for it to supported

[iainbeeston]

Sorry @JonRowe - can I please clarify something?

rspec-core needs updating even for multi-process parallelisation, right? (not just multi-threaded parallelisation as described in rspec/rspec-core#1254, which I would assume is more challenging)

[JonRowe]

The main rspec-core runner doesn't support parallelisation, so it has no means of collating results from workers, be they threads or processes. Process based parallelisation is simpler than thread based due to well, the GIL on MRI rubies and lack of 100% thread safety within the other gems. (I'm fairly certain for example that there are threaded bugs lurking within the mocks code).

My ideal plan for taking us to parallelisation is to use the "fork" model that the bisect runner uses, maybe with a fall back to a "shell" model (essentially processes) but I haven't had the time to work much on it.

[krtschmr]

to summarize: regardless if thread or processes: rspec-core isn't supporting it yet, so if we run a rails 6.0 application and use rspec, we won't benefit from it. correct?

[JonRowe]

The helpers are there to support parallelisation, rspec on its own doesn't support parallelisation so you'll need an extension gem to benefit from it.

[p8]

My ideal plan for taking us to parallelisation is to use the "fork" model that the bisect runner uses, maybe with a fall back to a "shell" model (essentially processes) but I haven't had the time to work much on it.

@JonRowe Would you copy all bisect code and renaming the namespace to parallel and make the required changes?

[JonRowe]

No I'm writing it from first principles using the bisect runner as a guide and some other external gems.

[wwahammy]

@JonRowe thanks for taking this on! Is there anything we can do to help this along? Not sure I'm qualified to write any of the runner but if there's something, please let us know!

[courtsimas]

Hey guys! What's the latest on this?

[JonRowe]

Still awaiting finishing the core work

[pirj]

Since Support Multithread Execution ticket on rspec-core doesn't seem to be a good fit, let me do a brain dump of tools that allow for parallelization of RSpec suite execution here:

• https://github.com/skroutz/rspecq - (added 31 July 2020) ci-queue inspired, Redis
• https://github.com/Shopify/ci-queue - RSpec support is deprecated, but not planned for removal, just not actively maintained; no support for before(:context)
• https://github.com/tmm1/test-queue - quite well known
• https://github.com/sandro/specjour - uses Bonjour to find worker processes
• https://github.com/cookpad/rrrspec - unmaintained for a year
• https://github.com/conversation/rspec-queue
• https://github.com/grosser/forking_test_runner
• https://github.com/paraspec/paraspec - just found it recently, didn't investigate
• https://github.com/lonelyplanet/queuecumber - the name says it all
• https://github.com/grosser/parallel_tests - does not aggregate reports. serves as a base for some other tools in this list
• https://github.com/ArturT/knapsack - splits the suite into equal'ish parts basing on previous runs' results. provides a dynamic queue for distributed rspec processes in the commercial version, but doesn't have on-premise option AFAIK
• https://github.com/aslakhellesoy/rspec-distributed - quite old. uses DRb, so can accumulate build results not only from several processes running on different CPU cores, but also from several machines. seems abandoned
• at my previous company, I was working on an in-house tool, but it's still not open-sourced unfortunately
• ... I keep bumping into new ones each and every month

All or most of those tools allow for parallelized execution in processes, arranging work with several databases via more or less common TEST_ENV_NUMBER env var.

With such a variety of tools, it seems that parallel testing of Rails applications with RSpec is not really an unsolved problem.
On my last project, we were running a dozen machines each running eight RSpec processes, that allowed to reduce spec suite wall clock run time 50x.
It's not by the way constrained by the usage of Rails 6, works equally in Rails 4.2+.

Basing on the efforts that had to be spent on building and maintaining such a tool while working full-time, I wouldn't really expect that a comparable tool to appear out of thin air in RSpec distribution and to immediately become better than everything else on the market.

[jtoy]

this will be awesome with rails 6+

[LandonSchropp]

@pirj I recently attempted to improve our CI testing using some of the tools you mentioned above, and I didn't have much luck. I didn't test all of them, but I did test the ones that were the most popular, and none really seemed to meet what we're looking for.

In an ideal world, a solution for this would do the following:

• If multiple cores are available, automatically parallelize the specs. This would have to be configurable.
• Automatically aggregate the results into a single, readable output stream using the provided formatter.
• Continue to work with coverage tools like SimpleCov.

My understanding of the Rails feature is that it gets us more of the way there. This still feels like an unsolved problem to me for Rspec. I believe Jest does something like this at the file level in JavaScript. It would be awesome to see.

[pirj]

@LandonSchropp I have bad news for you if you really need to aggregate the results into a single standard RSpec report. It's a hard task with a number of non-obvious edge cases. We've solved it on my previous project, at least for those cases that we've faced. The report is gathered from different workers, spread over different processes or even machines. Unfortunately, it's not open-sourced yet.
If you're using a JUnit-compatible XML-based reporter, combining the results in the end for CI to consume shouldn't be a big deal though.

As to my memory, we had no big issues with aggregating SimpleCov results, but that worked before I joined and I didn't work on this part.

Auto-scaling to multiple available cores is a trivial task. Once you (depending on your system flavour and CPU type) get the number of processors, you're able to run rake db:test:create and rspec by prefixing them with a TEST_ENV_NUMBER, and you're done. Some prefer leaving one core, but you may play around and run 2x more parallel processes than you have cores available. It depends on your tests what will yield the best results. Typically this is all done it a 20 LoC shell script.

Another non-trivial task is to evenly distribute your specs across those parallel processes. If you don't, you risk that one of the processes will be running for a significant amount of time, while other cores will finish already and will be idle. It is solved in Knapsack Pro and AFAIR in rrrspec.
The key is to keep the execution time of the specs, and feed specs to RSpec runner dynamically, one by one. Most time-consuming specs changed specs since the last run, and newly introduced specs are run first to reduce the unevenness of the distribution.
Frankly, I don't have any proof that this queuing system is faster than pre-divided execution on a pretty large suite.
Don't divide the spec into individual examples if you're using let_it_be, or some other form of before(:context) with a transaction rollback. The expensive setup will multiply in this case.

My understanding of the Rails feature is that it gets us more of the way there.

Honestly, I'm not so sure about that. A major flaw I could spot is that Rails doesn't detect if a worker died and did not finish all the workload it was supposed to execute. A better way is to have a timeout and consider worker as dead and spread over its remaining workload among alive workers. This is also solved in my previous project.

[LandonSchropp]

@pirj Thanks for the detailed reply!

Sorry if my post sounded like I was trivializing the amount of work involved in this. That definitely wasn't my intention. My goal in my last message was more to enumerate my team's use case in the hopes that it could influence the future design of this feature. I'm sure that a refactor along these lines is a massive undertaking.

As to my memory, we had no big issues with aggregating SimpleCov results, but that worked before I joined and I didn't work on this part.

I'll have to take a second look at that. When I was investigating, it wasn't clear to me how I'd go about that. I found one plugin for integrating SimpleCov with parallel-tests, but it said it was only compatible with CircleCI, and we're using Codeship, so I didn't explore it further. If you have any hints you could share, I'd appreciate it.

Auto-scaling to multiple available cores is a trivial task. Once you (depending on your system flavour and CPU type) get the number of processors, you're able to run rake db:test:create and rspec by prefixing them with a TEST_ENV_NUMBER, and you're done.

Yep, that's what we're currently doing. In my laptop, I've got 8 cores, each with hyperthreading, so that spins up 15 processes to run on. With that many simultaneous tests running, you can see how managing output would be helpful. I hadn't looked into aggregating JUnit XML compatible results. Do you have an example of something like that with Rspec you could point me to? Maybe that would help solve some of our problems. Our CI processes have four cores available, so it's somewhat less of a concern there.

Frankly, I don't have any proof that this queuing system is faster than pre-divided execution on a pretty large suite.

That's roughly my experience as well. We have a few specs in our application are that are slower than others, but for the most part the costs of individual specs are amortized over our files. Based on my observations of using Jest, it looks like it breaks the spec files into a queue, and then runs each file as a worker becomes available. In our use case, pre-splitting the specs has worked fine.

When I was exploring Knapsack, I actually felt like it was trying to solve a different problem than what we were experiencing. As you said, for us, it was less about distribution and more about taking advantage of multiple cores locally and in our CI environment.

A major flaw I could spot is that Rails doesn't detect if a worker died and did not finish all the workload it was supposed to execute. A better way is to have a timeout and consider worker as dead and spread over its remaining workload among alive workers. This is also solved in my previous project.

I can't speak to that. I'm sure you're absolutely correct, and I agree with you that the behavior you're describing sounds better than what's being done by the Rails runner.

What I meant more from my comment is that what's advertised from the Rails runner is more what we're after. It advertises that it automatically detects the number of cores on the machine, creates the test databases for us, and then splits up our tests and runs them. We love Rspec and will continue to use it over Minitest, but it would be awesome if there were an analog to the parallel features with Rspec that was as easy to use as the Rails solution.

Thanks again for your detailed reply. I really appreciate it!

[rabidaudio]

I can't speak to the rest of it, but as for merging JUnit reports, I've done that before in a different context. The file format is pretty simple, it would just be a matter of a little Nokogiri code to open each report and output a bigger report with the contents of each. Just have each rspec process output in JUnit format and then handle the aggregation and printing of results yourself

[SamSaffron]

@JonRowe Discourse would be happy to fund some work here if you or any other contributors are looking to build it.

Some requirements from me :)

• Dedicated gem, plug-and-play into your existing rspec rails app. Must be drop in replacement for Discourse.

• Fork per testing process

• Pull vs push model. (our bin/turbo_rspec is still the old push model, it means that some workers can be doing nothing while other workers have backlogs of work)... instead similar to @tmm1's test_queue master process has a queue of tests and child processes just ask it for the next test to run via an IO.pipe or some other cheap transport.

• Non interleaved output like Discourse's bin/turbo_rspec

• Long term also offers discourse bin/rake autospec which is basically a interruptible guard.

If anyone want to work on this and would like the work funded, DM me via Twitter. @SamSaffron

[JonRowe]

My plan is to build support for parallelism into rspec-core, which solves many of the problems with plug'n'play, both as our own option, and the ability to farm work out to other plugins. Fork is absolutely on those planes. I very much like the idea of the master rspec process controlling the queue like that, as it allows much easier non interleaved output, which was very much essential.

From there rspec-rails would than integrate the Rails system test helpers for their database support etc

I hadn't thought about the possibility of such a model being a long running process, and reloading / rerunning specs, thats got a set of complications all on its own (as I'm sure you know!)

[thbar]

@JonRowe it must be noted too that parallelism in Rspec will also give a strong acceleration to all the "TDD devops" tools which happen to be based on RSpec (like InSpec and ServerSpec). Just a side-dish, but a very nice one, and definitely useful to the reputation of Ruby as a whole ^_^. Given that, maybe Chef could also be willing to provide some funding in that area (I'm not working there, but just making a wild guess!).

[wwahammy]

@JonRowe @SamSaffron My company, CommitChange, would be able to add a small amount to the parallel testing pot. It'd be well under $1000 (we just don't have the resources to spend a lot more) but it's something if it'd help. I totally get if this isn't enough to bother but there's likely many smaller users who might be able to put in a few hundred.

[SamSaffron]

thanks everyone, I have been chatting privately to @JonRowe and it looks like Jon may have some time for this project, stay tuned 🎊

[griou]

Hey 👋
Love RSpec ❤️ but its main weakness to me it's that it has no built-in parallelization 😢 It leads many teams to run the full tests in the CI and not on their local environment, which I believe is not the best practice in 2022 compared to other languages or test frameworks. This thread is opened for almost 3 years, small questions:

• Was anyone able to accomplish parallel testing with RSpec and rails6
• Will RSpec support built-in parallel testing this year 😓

I guess it's not an easy topic 🙏 ❤️

[pirj]

this year

No promises. This depends on contributors mostly. RSpec core team has limited resources and is working on releasing support for Rails 7, trying to get RSpec 4 out of the door, and a number of housekeeping tasks.

I encourage you to take a stab at it, and report how far you went through, what went wrong. We can adjust RSpec Core/Mocks as needed to support in-process concurrency.

[bmulholland]

I think where this was left off is that the result of "take a stab at it" is https://github.com/ioquatix/turbo_test/, and so next steps for anyone who wants parallel test support (me included :) ) is to implement that gem and report any issues. Is that right?

If turbo_test gets enough adoption and validation, would that approach then be folded into rspec?

[pirj]

Checking Rails testing guide:

forking processes is the default method

Process forking approach has been available for running RSpec in parallel in the local development for a long time already. Check #2104 (comment) for a list. There are numerous blog posts how to set it up, including for Rails.

This shifts the "inherent deficiency, obviously a downside" argument into "is not available out of the box, needs additional setup". And, from my perspective, shifts its priority a bit down.

Should we embrace and absorb, bundle or reinvent one of those tools?
It comes with a burden to support it.

Should we work on supporting in-process concurrency (threads, fibers, ractors, ... whatever comes next)?
Basically, the same answer as above.

Should we give our preference and advertise one of the tools in our documentation, optionally provide it as part of our rails generate rspec:install template?
As you've seen, turbo_test development was very active at some point, and then suddenly stopped. This happened to other tools, too. We have no control over that.

I would be happy to see a streamlined approach, but it's unlikely to happen without outstanding contributors' dedication to develop and spread the word. The tool should really stand out, as the currently existing ones work in most cases, too.

[supairish]

@griou We run our rspec/capybara test suite using https://github.com/grosser/parallel_tests It works well, you just have to make some different config adjustments for things that run outside of rails during the test suite run, i.e. capybara ports, database selection (per process), elasticsearch indices (per process), redis, etc... See the wiki for examples https://github.com/grosser/parallel_tests/wiki

[Unknown-Guy]

@pirj I think I have to add another perspective here. Been around Rails projects since 2.3 was latest and greatest. Up until now for every new project we went with RSpec. But now starting on a new project there is really big incentive to switch - it's included out of the box, automatic and will be supported.

So it is not only missing feature, but becoming kind of branding issue.

In-process concurrency would be great, but maybe first step is to just provide up to date list of existing tool and provide some options in install template. The template option might be bridge to other tool owners to iron out some integration issues and simply drive interest to them.

[ilyazub]

https://github.com/serpapi/turbo_tests author here. Shameless plug, but it's available as a gem and works as a replacement for the parallel_tests. We use it for 1.5 years in CI and in development every day.

PS. All I did is copy-pasted part of the source code from Rubygems and Discourse, adjusted it a bit, and packaged it into a gem (#2104 (comment)).

[bmulholland]

FWIW, https://github.com/serpapi/turbo_tests worked out of the box (added gem, ran bundle exec turbo_tests) and halved my dev machine test suite runtime. Everything passed first time (admittedly a very simple test suite).

By contrast, https://github.com/ioquatix/turbo_test required a bunch of setup steps (ioquatix/turbo_test#6) and is now throwing an error when I try it again.

[pirj]

copy-pasted part of the source code from ... Discourse

MIT vs GPLv2 😱
@SamSaffron What do you think?

Worked perfectly fine as a drop-in replacement/wrapper for parallel_tests. Good job, @ilyazub !

Before:

bundle exec rspec spec/models/  646.73s user 251.24s system 86% cpu 17:14.30 total

After:

bundle exec turbo_tests spec/models  966.81s user 339.88s system 216% cpu 10:04.89 total

What was missing (and what Rails provides with its default testing framework):

Active Record automatically handles creating a database

Without createdb myproject2 etc it didn't work. I've seen rails db:test:prepare, but db:create wasn't apparently called. I blame this on parallel_tests.

Spreading out tests evenly, dynamically (with a queue), and considering individual specs' CPU/IO ratio is out of comparison and is on par with Rails.

[jkeen]

FWIW, https://github.com/serpapi/turbo_tests worked out of the box (added gem, ran bundle exec turbo_tests) and halved my dev machine test suite runtime.

Same! Been following this thread since 2019 and this week it paid off—thanks @ilyazub!

[ilyazub]

@bmulholland @pirj @jkeen Thanks for your kind words!

---

copy-pasted part of the source code from ... Discourse

MIT vs GPLv2 scream @SamSaffron What do you think?

I guess I can change the license from MIT to GPLv2 and add copyright headers if needed. I've used the MIT license because that's what bundle gem turbo_tests provided.

---

What was missing (and what Rails provides with its default testing framework):

Active Record automatically handles creating a database

Without createdb myproject2 etc it didn't work. I've seen rails db:test:prepare, but db:create wasn't apparently called. I blame this on parallel_tests.

Yes, parallel_tests requires manual invocation of rake parallel:setup. But I've incorrectly written in the gem description about how the parallel_tests gem is used. Please don't blame parallel_tests for that :-)

@pirj Thank you for mentioning the automatic DB creation in Rails. I've opened an issue in the turbo_tests repo about an automatic setup of the database.

---

Btw, ActiveRecord::TestCase uses DRb for IPC. At some point, I'd like to try using DRb in turbo_tests.

[ArturT]

I guess I can change the license from MIT to GPLv2 if needed. I've used the MIT license because that's what bundle gem turbo_tests provided.

Most likely you have to use GPLv2. Only users of your gem might have some problem if they don't check the license is GPLv2.
This reminds me of last year mess in the Rails community https://news.ycombinator.com/item?id=26566925

[pirj]

users of your gem might have some problem if they don't check the license is GPLv2

Only if they decide to supply a modified version of turbo_tests as an integral part of their proprietary software to another party. To my best knowledge, GPL does not apply any restrictions on using GPL-licensed software for CI or local development, commercial or otherwise, modified or not.

The Rails community incident is the reason why I mention GPL vs MIT above:

GPL header removed.
The license that you're shipping mimemagic under (MIT) isn't compatible with shared-mime-info's.

[SamSaffron]

Happy for the code that was extracted from discourse to be treated as MIT!

…

On Sat, 29 Jan 2022 at 7:21 pm, Phil Pirozhkov ***@***.***> wrote: users of your gem might have some problem if they don't check the license is GPLv2 Only if they decide to supply a modified version of turbo_tests as an integral part of their proprietary software to another party. To my best knowledge, GPL does not apply any restrictions on using GPL-licensed software for CI or local development, commercial or otherwise, modified or not. The Rails community incident <mimemagicrb/mimemagic#97 (comment)> is the reason why I mention GPL vs MIT above <#2104 (comment)> : GPL header removed. The license that you're shipping mimemagic under (MIT) isn't compatible with shared-mime-info's. — Reply to this email directly, view it on GitHub <#2104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABIXNRHDZZYR7UTKL6KHLUYOPQHANCNFSM4HCVUOKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pirj]

Thanks a lot, @SamSaffron ! ❤️

@ilyazub I suppose you can consider this written permission to use parts of GPL-licensed Discourse code in your MIT-licensed turbo_tests.

10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.

[ioquatix]

Great discussion, I'm really excited to see people passionate about testing!

Just reading through some of the comments here, it seems like we need a better model for parallel testing. From what I see, I think turbo_test (my hack/implementation) has the right approach by separating out the worker setup into an explicit step, given that serpapi/turbo_tests#18 seems to be exactly the same issue. What we put into that worker setup depends on your app, db setup, and Rails.

Regarding parallelism, I think there are enough issues in Rails test harness that some improvements there would be useful: better handling of TEST_ENV_NUMBER would be an obvious one, along with better db migrations using a proper interface rather than shelling out to rake. RSpec also has a ton of global state which makes it hard to work with (think hacks to reset global state after each worker runs a test), so there is also work to be done here.

To this end, I decided to try implementing a green-field parallel test runner and ended up with https://github.com/ioquatix/sus which I'm now using in a handful of projects. It's designed from the ground up for multi-thread parallelism by ensuring all tests can be run in total isolation (i.e. the semantic model is built from the ground up for isolation). Interestingly enough, I've been toying with an idea to host other test frameworks within sus, i.e. hosting Minitest/RSpec tests within the sus parallel runner.

Maybe rspec-core can follow a similar model to sus but a large amount of rspec would need to be rewritten, essentially the entire output/notification system (it's unsafe/hard to parallelise/global state) and the test statistics capture (currently also seems like global state). @JonRowe correct me if I'm wrong please. Separation between running a set of tests and the test model itself would also be really useful. sus exposes tests as an hierarchy of callable objects and you can execute them any way you want, so this gives a lot of power for different execution models - multi-thread - multi-process - multi-host - etc. Tests have (guaranteed) consistent unique identifiers which you can transit across those boundaries (i.e. hosts) and reload in a target process which ensures that these models all work as expected (doesn't depend on load order etc).

After playing around in this space for a while, I think what we want is this:

1. A way to load a callable hierarchy of tests with unique identifiers from a specific test framework e.g. sus, minitest, rspec, etc.
2. A command to execute this callable hierarchy and report failures.

sus unfortunately combines both of these things into a single gem, but it's been on my todo list to explore having a clean separation between these two, and write an adapter for rspec. If we adopt this approach generally, it should be far easier to define a clean model for multi-thread/process/host parallelism that works generally across a wide range of test definitions.

[Unknown-Guy]

Shelling out on MRI is not that bad. Did a quick test in our project with turbo_test - jRuby and Open3.popen3 is not the best friends in this case:

bin/turbo_tests spec/models  2460.21s user 748.40s system 356% cpu 15:00.20 total

vs

bin/rspec spec/models  661.43s user 33.36s system 70% cpu 16:31.66 total

[ilyazub]

@Unknown-Guy Yes, a one-minute speed-up could be better. How many *_spec.rb files do you have?

[Unknown-Guy]

@ilyazub there's 277 files there, 4425 examples

Edit: think the startup is what eats the time here, shelling out and starting another jruby process is very expensive. Maybe this could work together with something like Theine.

Just wanted to add that that the approach with shelling out has pretty big drawback on jRuby.

[ryanong]

Looks like a new tool for rspec parallel testing https://github.com/briandunn/flatware

[ilyazub]

Looks like a new tool for rspec parallel testing https://github.com/briandunn/flatware

@ryanong Thank you for suggesting it! It uses drb and I love the flatware's code structure and quality.

Unfortunately, it's slower than turbo_tests (and now I'm not sure about re-writing it using drb).

$ time bundle exec flatware rspec -w 8 ./spec/search/google/
real    6m7.343s
user    0m10.354s
sys     0m1.710s

$ time bundle exec turbo_tests ./spec/search/google/
real    4m56.069s
user    23m9.252s
sys     0m11.208s

Tested with Ruby 2.7.2 on 8 core i7-6700HQ CPU @ 2.60GHz.

$ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

There're 196 spec files (2,1K examples) in the directory I've used to compare the speed.

find ./spec/search/google/ -name '*_spec.rb' -type f | wc -l
196

find ./spec/search/google/ -name '*_spec.rb' -type f -print0 | xargs -0 grep -o 'it ' | wc -l
2149

P.S. Maybe it's possible to parallel execution of examples instead of spec files using flatware 🤔 💡

[mvz]

For me, flatware was faster than turbo_tests, so best to benchmark this I think.

[jdelStrother]

Not very rigorous, but FWIW on my M1 MBP (8-core, 32GB), running our Rails project's 4260 examples across 526 spec files:

tool	times
non-parallel rspec	194.53 real 137.02 user 27.78 sys
parallel_tests	60.51 real 209.87 user 47.46 sys
turbo_test	60.26 real 201.67 user 44.91 sys
flatware	55.53 real 183.73 user 35.53 sys

I ran each a handful of times and reported the best time (which is admittedly unfairly biased in favour of parallel_tests, which would often have by far the worst of the parallel runners' times due to poorly balanced specs)

flatware does seem consistently faster for me, although it has a few rough edges with things like not being able to interrupt the tests with ctrl-c, the extra complications of a flatware-sink background process, and an unexplained single test failure I haven't figured out the cause of yet.

[briandunn]

@jdelStrother Just released a new version - would love to address these concerns could you open an issue?

[jdelStrother]

@briandunn seems pretty good now! It behaves properly with ctrl-c, and I don't seem to be seeing the random test failure from last time I tried.