Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MultipleMemoizedHelpers cop #863

Merged
merged 9 commits into from Aug 9, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Expand Up @@ -11,6 +11,7 @@
* Improve `RSpec/NestedGroups`, `RSpec/FilePath`, `RSpec/DescribeMethod`, `RSpec/MultipleDescribes`, `RSpec/DescribeClass`'s top-level example group detection. ([@pirj][])
* Add detection of `let!` with a block-pass or a string literal to `RSpec/LetSetup`. ([@pirj][])
* Add `IgnoredPatterns` configuration option to `RSpec/VariableName`. ([@jtannas][])
* Add `RSpec/MultipleMemoizedHelpers` cop. ([@mockdeep][])

## 1.42.0 (2020-07-09)

Expand Down Expand Up @@ -542,3 +543,4 @@ Compatibility release so users can upgrade RuboCop to 0.51.0. No new features.
[@elliterate]: https://github.com/elliterate
[@mlarraz]: https://github.com/mlarraz
[@jtannas]: https://github.com/jtannas
[@mockdeep]: https://github.com/mockdeep
7 changes: 7 additions & 0 deletions config/default.yml
Expand Up @@ -394,6 +394,13 @@ RSpec/MultipleExpectations:
VersionChanged: '1.21'
StyleGuide: https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleExpectations

RSpec/MultipleMemoizedHelpers:
Description: Checks if example groups contain too many `let` and `subject` calls.
Enabled: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️
I personally don't have many doubts that this cop should be enabled by default. Open for the discussion, though.

Just in case, we only have a handful of disabled cops, and we encourage (not documented) to tune the configuration to meet the project style, not bend all projects to our defaults.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with having it enabled, with AllowSubject true, by default

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about Enabled: pending? This cop broke my builds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexWayfer We didn't formally agree to adhere to this policy yet, neither we're sure that it works for extensions.
We'll do our best to make it work and to introduce new cops between 2.0 and 3.0 as pending.
Meanwhile - please accept my apologies.

AllowSubject: true
Max: 5
StyleGuide: https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleMemoizedHelpers

RSpec/MultipleSubjects:
Description: Checks if an example group defines `subject` multiple times.
Enabled: true
Expand Down
148 changes: 148 additions & 0 deletions lib/rubocop/cop/rspec/multiple_memoized_helpers.rb
@@ -0,0 +1,148 @@
# frozen_string_literal: true

module RuboCop
module Cop
module RSpec
# Checks if example groups contain too many `let` and `subject` calls.
#
# This cop is configurable using the `Max` option and the `AllowSubject`
# which will configure the cop to only register offenses on calls to
# `let` and not calls to `subject`.
#
# @example
# # bad
# describe MyClass do
# let(:foo) { [] }
# let(:bar) { [] }
# let!(:baz) { [] }
# let(:qux) { [] }
# let(:quux) { [] }
# let(:quuz) { {} }
# end
#
# describe MyClass do
# let(:foo) { [] }
# let(:bar) { [] }
# let!(:baz) { [] }
#
# context 'when stuff' do
# let(:qux) { [] }
# let(:quux) { [] }
# let(:quuz) { {} }
# end
# end
#
# # good
# describe MyClass do
# let(:bar) { [] }
# let!(:baz) { [] }
# let(:qux) { [] }
# let(:quux) { [] }
# let(:quuz) { {} }
# end
#
# describe MyClass do
# context 'when stuff' do
# let(:foo) { [] }
# let(:bar) { [] }
# let!(:booger) { [] }
# end
#
# context 'when other stuff' do
# let(:qux) { [] }
# let(:quux) { [] }
# let(:quuz) { {} }
# end
# end
#
# @example when disabling AllowSubject configuration
#
# # rubocop.yml
# # RSpec/MultipleMemoizedHelpers:
# # AllowSubject: false
#
# # bad - `subject` counts towards memoized helpers
# describe MyClass do
# subject { {} }
# let(:foo) { [] }
# let(:bar) { [] }
# let!(:baz) { [] }
# let(:qux) { [] }
# let(:quux) { [] }
# end
#
# @example with Max configuration
#
# # rubocop.yml
# # RSpec/MultipleMemoizedHelpers:
# # Max: 1
#
# # bad
# describe MyClass do
# let(:foo) { [] }
# let(:bar) { [] }
# end
#
class MultipleMemoizedHelpers < Base
include ConfigurableMax
include RuboCop::RSpec::Variable

MSG = 'Example group has too many memoized helpers [%<count>d/%<max>d]'

def on_block(node)
return unless spec_group?(node)

count = all_helpers(node).uniq.count

return if count <= max

self.max = count
add_offense(node, message: format(MSG, count: count, max: max))
pirj marked this conversation as resolved.
Show resolved Hide resolved
end

def on_new_investigation
@example_group_memoized_helpers = {}
end

private

attr_reader :example_group_memoized_helpers

def all_helpers(node)
[
*helpers(node),
*node.each_ancestor(:block).flat_map(&method(:helpers))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In deeply nested code, or very long spec files, I imagine this line has the potential to be a performance hotspot, right? We might want to do some memoization for each ancestor.

I’d need to do some measurement to figure out if it’s really a problem or not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bquorning Fair enough.
Ran it the following against gitlabhq and discourse:

$ bundle exec bin/rubocop-profile --only RSpec --force-default-config  --require rubocop-rspec ../real-world-rspec/gitlabhq ../real-world-rspec/discourse
$ stackprof tmp/stackprof.dump --method 'RuboCop::Cop::RSpec'

The results (cleaned up, no idea how to get a more compact output by default):

RuboCop::Cop::RSpec::Base#rspec_pattern (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/base.rb:47)
  samples:  25142 self (4.1%)  /   28918 total (4.8%)
RuboCop::Cop::RSpec::AnyInstance#disallowed_stub (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/any_instance.rb:28)
  samples:  1721 self (0.3%)  /   2268 total (0.4%)
RuboCop::Cop::RSpec::Base#relevant_rubocop_rspec_file? (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/base.rb:43)
  samples:  1555 self (0.3%)  /   30473 total (5.0%)
RuboCop::Cop::RSpec::ImplicitExpect#implicit_expect (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/implicit_expect.rb:33)
  samples:  1478 self (0.2%)  /   2480 total (0.4%)
RuboCop::Cop::RSpec::MessageSpies#message_expectation (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/message_spies.rb:38)
  samples:  1392 self (0.2%)  /   1758 total (0.3%)

[45 items skipped]

RuboCop::Cop::RSpec::MultipleMemoizedHelpers#on_block (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/multiple_memoized_helpers.rb:92)
  samples:   214 self (0.0%)  /   20335 total (3.4%)

[4 items skipped]

RuboCop::Cop::RSpec::MultipleMemoizedHelpers#all_helpers (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/multiple_memoized_helpers.rb:105)
  samples:   179 self (0.0%)  /   18647 total (3.1%)

[38 items skipped]

RuboCop::Cop::RSpec::MultipleMemoizedHelpers#allow_subject? (/Users/pirj/source/rubocop-rspec/lib/rubocop/cop/rspec/multiple_memoized_helpers.rb:135)
  samples:    51 self (0.0%)  /    161 total (0.0%)

[...]

  callees (18 total):
      18  (  100.0%)  RuboCop::Cop::Base.inherited

I'm not certain how to interpret this, is it the first or the second percentage that is important.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, we should determine if MultipleMemoizedHelpers is slower than the other cops. If its speed is average, there’s no need to dig further.

I ran the same command as yours, but on another spec suite. Next, I search the stackprof result for all on_* method calls on RSpec cops and order them by number of samples (20335 for on_block in your example above)

stackprof tmp/stackprof.dump --method 'RuboCop::Cop::RSpec.*#on_.*' |
  grep -B1 'samples.* total' |
  paste -d " "  - - - |
  sort -k8 -n |
  awk '{ print $8 " samples: " $1 }'

The final lines of my result is

485 samples: RuboCop::Cop::RSpec::RepeatedExample#on_block
517 samples: RuboCop::Cop::RSpec::RepeatedDescription#on_block
577 samples: RuboCop::Cop::RSpec::MultipleExpectations#on_block
844 samples: RuboCop::Cop::RSpec::SubjectStub#on_top_level_group
960 samples: RuboCop::Cop::RSpec::DescribedClass#on_block
1132 samples: RuboCop::Cop::RSpec::MultipleMemoizedHelpers#on_block

So it seems that SubjectStub, DescribedClass and MultipleMemoizedHelpers are significantly slower than our other cops. (Of course, configuration and the nature of the specs may play a role here)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a trivial memoization.
Before
713.80s user 42.25s system 80% cpu 15:42.32 total
After
663.91s user 41.03s system 89% cpu 13:10.78 total

Before

7407 samples: RuboCop::Cop::RSpec::MultipleSubjects#on_block
7969 samples: RuboCop::Cop::RSpec::LetSetup#on_block
7998 samples: RuboCop::Cop::RSpec::RepeatedExample#on_block
8527 samples: RuboCop::Cop::RSpec::RepeatedDescription#on_block
9783 samples: RuboCop::Cop::RSpec::MultipleExpectations#on_block
12220 samples: RuboCop::Cop::RSpec::SubjectStub#on_top_level_group
13026 samples: RuboCop::Cop::RSpec::DescribedClass#on_block
27289 samples: RuboCop::Cop::RSpec::MultipleMemoizedHelpers#on_block <===

After

6297 samples: RuboCop::Cop::RSpec::NamedSubject#on_block
6305 samples: RuboCop::Cop::RSpec::MultipleSubjects#on_block
6686 samples: RuboCop::Cop::RSpec::RepeatedExample#on_block
6846 samples: RuboCop::Cop::RSpec::RepeatedDescription#on_block
6939 samples: RuboCop::Cop::RSpec::LetSetup#on_block
7592 samples: RuboCop::Cop::RSpec::MultipleMemoizedHelpers#on_block <===
8599 samples: RuboCop::Cop::RSpec::MultipleExpectations#on_block
10258 samples: RuboCop::Cop::RSpec::SubjectStub#on_top_level_group

It doesn't look like those results are very consistent, as other cops are shuffled. But at least MultipleMemoizedHelpers isn't the by far the worst offender anymore.

Is this good enough, @bquorning ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, sampling results aren’t always consistent. But moving from 27000 samples down to 7600 is a significant change for a relatively simple memoization. Thank you for taking the time to implement this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to do so, especially when it's that trivial.

]
end

def helpers(node)
@example_group_memoized_helpers[node] ||=
variable_nodes(node).map do |variable_node|
if variable_node.block_type?
variable_definition?(variable_node.send_node)
else # block-pass (`let(:foo, &bar)`)
variable_definition?(variable_node)
end
end
end

def variable_nodes(node)
example_group = RuboCop::RSpec::ExampleGroup.new(node)
if allow_subject?
example_group.lets
else
example_group.lets + example_group.subjects
end
end

def max
cop_config['Max']
end

def allow_subject?
cop_config['AllowSubject']
end
end
end
end
end
1 change: 1 addition & 0 deletions lib/rubocop/cop/rspec_cops.rb
Expand Up @@ -65,6 +65,7 @@
require_relative 'rspec/missing_example_group_argument'
require_relative 'rspec/multiple_describes'
require_relative 'rspec/multiple_expectations'
require_relative 'rspec/multiple_memoized_helpers'
require_relative 'rspec/multiple_subjects'
require_relative 'rspec/named_subject'
require_relative 'rspec/nested_groups'
Expand Down
2 changes: 1 addition & 1 deletion lib/rubocop/rspec/variable.rb
Expand Up @@ -8,7 +8,7 @@ module Variable
extend RuboCop::NodePattern::Macros

def_node_matcher :variable_definition?, <<~PATTERN
(send #rspec? #{(Helpers::ALL + Subject::ALL).node_pattern_union}
(send nil? #{(Helpers::ALL + Subject::ALL).node_pattern_union}
pirj marked this conversation as resolved.
Show resolved Hide resolved
$({sym str dsym dstr} ...) ...)
PATTERN
end
Expand Down
1 change: 1 addition & 0 deletions manual/cops.md
Expand Up @@ -64,6 +64,7 @@
* [RSpec/MissingExampleGroupArgument](cops_rspec.md#rspecmissingexamplegroupargument)
* [RSpec/MultipleDescribes](cops_rspec.md#rspecmultipledescribes)
* [RSpec/MultipleExpectations](cops_rspec.md#rspecmultipleexpectations)
* [RSpec/MultipleMemoizedHelpers](cops_rspec.md#rspecmultiplememoizedhelpers)
* [RSpec/MultipleSubjects](cops_rspec.md#rspecmultiplesubjects)
* [RSpec/NamedSubject](cops_rspec.md#rspecnamedsubject)
* [RSpec/NestedGroups](cops_rspec.md#rspecnestedgroups)
Expand Down
102 changes: 102 additions & 0 deletions manual/cops_rspec.md
Expand Up @@ -2119,6 +2119,108 @@ Max | `1` | Integer

* [https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleExpectations](https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleExpectations)

## RSpec/MultipleMemoizedHelpers

Enabled by default | Safe | Supports autocorrection | VersionAdded | VersionChanged
--- | --- | --- | --- | ---
Enabled | Yes | No | - | -

Checks if example groups contain too many `let` and `subject` calls.

This cop is configurable using the `Max` option and the `AllowSubject`
which will configure the cop to only register offenses on calls to
`let` and not calls to `subject`.

### Examples

```ruby
# bad
describe MyClass do
let(:foo) { [] }
let(:bar) { [] }
let!(:baz) { [] }
let(:qux) { [] }
let(:quux) { [] }
let(:quuz) { {} }
end

describe MyClass do
let(:foo) { [] }
let(:bar) { [] }
let!(:baz) { [] }

context 'when stuff' do
let(:qux) { [] }
let(:quux) { [] }
let(:quuz) { {} }
end
end

# good
describe MyClass do
let(:bar) { [] }
let!(:baz) { [] }
let(:qux) { [] }
let(:quux) { [] }
let(:quuz) { {} }
end

describe MyClass do
context 'when stuff' do
let(:foo) { [] }
let(:bar) { [] }
let!(:booger) { [] }
end

context 'when other stuff' do
let(:qux) { [] }
let(:quux) { [] }
let(:quuz) { {} }
end
end
```
#### when disabling AllowSubject configuration

```ruby
# rubocop.yml
# RSpec/MultipleMemoizedHelpers:
# AllowSubject: false

# bad - `subject` counts towards memoized helpers
describe MyClass do
subject { {} }
let(:foo) { [] }
let(:bar) { [] }
let!(:baz) { [] }
let(:qux) { [] }
let(:quux) { [] }
end
```
#### with Max configuration

```ruby
# rubocop.yml
# RSpec/MultipleMemoizedHelpers:
# Max: 1

# bad
describe MyClass do
let(:foo) { [] }
let(:bar) { [] }
end
```

### Configurable attributes

Name | Default value | Configurable values
--- | --- | ---
AllowSubject | `true` | Boolean
Max | `5` | Integer

### References

* [https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleMemoizedHelpers](https://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/MultipleMemoizedHelpers)

## RSpec/MultipleSubjects

Enabled by default | Safe | Supports autocorrection | VersionAdded | VersionChanged
Expand Down