New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single file skip parallel #10903
Single file skip parallel #10903
Conversation
77ea566
to
b7cd7ce
Compare
@@ -64,6 +64,10 @@ def aborting? | |||
# instances that each inspects its allotted group of files. | |||
def warm_cache(target_files) | |||
saved_options = @options.dup | |||
if target_files.length <= 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw, shouldn't this be the first line of the method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we make this the first line of the method then the ensure
clause needs to have a separate begin
block, otherwise it will replace @options
with nil. I opted for this way because it looked slightly cleaner to my eyes and needed less changes.
Would you like me to rewrite it to skip the .dup
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I didn't notice the ensure
. Probably it would have been nicer to have here if/else
instead of if
+ return
, although I assume RuboCop forced you to write the code like this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following passes tests and rubocop inspection:
# Warms up the RuboCop cache by forking a suitable number of RuboCop
# instances that each inspects its allotted group of files.
def warm_cache(target_files)
if target_files.length <= 1
puts 'Skipping parallel inspection: only a single file needs inspection' if @options[:debug]
else
begin
saved_options = @options.dup
puts 'Running parallel inspection' if @options[:debug]
%i[autocorrect safe_autocorrect].each { |opt| @options[opt] = false }
Parallel.each(target_files) { |target_file| file_offenses(target_file) }
ensure
@options = saved_options
end
end
end
Can you take a look at this test failure:
it's different than the flaky test failure from #10897. |
@dvandersluis I looked at the failure but it does not look like it should be caused by the changes in this MR. I tried reproducing the error on my machine but cannot get it to fail the test suite even with many test queue workers. 😐 |
Can you rebase on top of master? |
b7cd7ce
to
9f1750d
Compare
@koic Any idea why the Windows jobs are failing? This seems unrelated to the changes. |
This looks like an expected failure on Windows CI. Failures:
1) RuboCop::CLI options --parallel on Windows prints a warning
Failure/Error: expect($stderr.string).to include('Process.fork is not supported by this Ruby')
expected "" to include "Process.fork is not supported by this Ruby"
# ./spec/rubocop/cli/options_spec.rb:19:in `block (4 levels) in <top (required)>'
# ./spec/support/cli_spec_behavior.rb:26:in `block (2 levels) in <top (required)>'
# ./lib/rubocop/rspec/shared_contexts.rb:30:in `block (4 levels) in <top (required)>'
# ./lib/rubocop/rspec/shared_contexts.rb:30:in `chdir'
# ./lib/rubocop/rspec/shared_contexts.rb:30:in `block (3 levels) in <top (required)>'
# ./lib/rubocop/rspec/shared_contexts.rb:7:in `block (2 levels) in <top (required)>'
# tasks/spec_runner.rake:40:in `block in run_specs'
# tasks/spec_runner.rake:52:in `with_encoding'
# tasks/spec_runner.rake:36:in `run_specs'
# tasks/spec_runner.rake:165:in `block in <top (required)>'
Finished in 2 minutes 55.7 seconds (files took 4.16 seconds to load)
18732 examples, 1 failure, 15 pending
Failed examples:
rspec ./spec/rubocop/cli/options_spec.rb:17 # RuboCop::CLI options --parallel on Windows prints a warning https://github.com/rubocop/rubocop/runs/7945357268 The message Probably this message is no longer printed because the test only has one inspection target file. |
I'll have a look at it this weekend. |
6504ec2
to
3e713bd
Compare
@@ -0,0 +1 @@ | |||
* [#1003](https://github.com/rubocop/rubocop/pull/10903): Skip forking off extra processes for parallel inspection when only a single file needs to be inspected. ([@wjwh][]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you update the PR number? 😅
* [#1003](https://github.com/rubocop/rubocop/pull/10903): Skip forking off extra processes for parallel inspection when only a single file needs to be inspected. ([@wjwh][]) | |
* [#10903](https://github.com/rubocop/rubocop/pull/10903): Skip forking off extra processes for parallel inspection when only a single file needs to be inspected. ([@wjwh][]) |
I commented on the changelog entry, anyway overall it looks good. Can you squash your commits into one? |
1938fb9
to
7f1e015
Compare
It took us a while, but we made it to the finish line. :-) Thanks! |
Thanks for the help and guidance! |
This MR makes it so that if there is only a single file to be inspected, we do not split off extra worker processes to do the work. This is because doing the work in a separate process can never be faster than doing it inline. I benchmarked this change on my dev machine (with 6 physical cores) and it saves 40-50 ms per run (only if only a single file is inspected of course, otherwise it will take as long as before).
I did not know what to update the test with, so I just made it expect the new text. Let me know if this should be different.
(This is a fixed version of #10889, where I had messed up the rebasing)
Before submitting the PR make sure the following are checked:
Commit message starts with[Fix #issue-number]
(if the related issue exists).master
(if not - rebase it).bundle exec rake default
. It executes all tests and runs RuboCop on its own code.{change_type}_{change_description}.md
if the new code introduces user-observable changes. See changelog entry format for details.