From 94eca165ce5456c3631565c1fa4a6b9a63e70c06 Mon Sep 17 00:00:00 2001 From: Dana Sherson Date: Sun, 8 Mar 2020 03:18:13 +1100 Subject: [PATCH] Add subprocess handling to simplecov (#881) * Add subprocess handling to simplecov Using Process.fork (whether using the Parallel gem or directly) creates code that was invisible to SimpleCov. by starting SimpleCov within the subprocess with its own command name and etc we can see that code :) This also adds documentation for what to do when using Process.spawn or similar. fixes: #414 * Subprocesses is off by default Can be enabled with enable_for_subprocesses Also moved the testing to a feature test * Fix test to work for ruby 2.4 also * Have spawn doc match closer to spec * Remove unnecessary = Also Gem::Version.new comparisons please. Also don't run this test on jruby, Process.fork is NotImplementedError --- CHANGELOG.md | 6 +++ README.md | 47 +++++++++++++++++++ .../config_enable_for_subprocesses.feature | 37 +++++++++++++++ features/support/env.rb | 10 ++-- lib/simplecov.rb | 2 + lib/simplecov/configuration.rb | 46 ++++++++++++++++++ lib/simplecov/process.rb | 19 ++++++++ test_projects/subprocesses/.simplecov | 5 ++ .../subprocesses/.simplecov_spawn.rb | 5 ++ test_projects/subprocesses/lib/command | 7 +++ .../subprocesses/lib/subprocesses.rb | 21 +++++++++ .../subprocesses/spec/simple_spec.rb | 7 +++ test_projects/subprocesses/spec/spawn_spec.rb | 10 ++++ .../subprocesses/spec/spec_helper.rb | 6 +++ 14 files changed, 225 insertions(+), 3 deletions(-) create mode 100644 features/config_enable_for_subprocesses.feature create mode 100644 lib/simplecov/process.rb create mode 100644 test_projects/subprocesses/.simplecov create mode 100644 test_projects/subprocesses/.simplecov_spawn.rb create mode 100755 test_projects/subprocesses/lib/command create mode 100644 test_projects/subprocesses/lib/subprocesses.rb create mode 100644 test_projects/subprocesses/spec/simple_spec.rb create mode 100644 test_projects/subprocesses/spec/spawn_spec.rb create mode 100644 test_projects/subprocesses/spec/spec_helper.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 367942ce..434952ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +Unreleased +========== + +## Enhancements +* observe forked processes (enable with SimpleCov.enable_for_subprocesses) + 0.18.5 (2020-02-25) =================== diff --git a/README.md b/README.md index 196c8d3c..d92fc71b 100644 --- a/README.md +++ b/README.md @@ -614,6 +614,53 @@ namespace :coverage do end ``` +## Running simplecov against subprocesses + +`SimpleCov.enable_for_subprocesses` will allow SimpleCov to observe subprocesses starting using `Process.fork`. +This modifies ruby's core Process.fork method so that SimpleCov can see into it, appending `" (subprocess #{pid})"` +to the `SimpleCov.command_name`, with results that can be merged together using SimpleCov's merging feature. + +To configure this, use `.at_fork`. + +```ruby +SimpleCov.enable_for_subprocesses true +SimpleCov.at_fork do |pid| + # This needs a unique name so it won't be ovewritten + SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})" + # be quiet, the parent process will be in charge of output and checking coverage totals + SimpleCov.print_error_status = false + SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter + SimpleCov.minimum_coverage 0 + # start + SimpleCov.start +end +``` + +NOTE: SimpleCov must have already been started before `Process.fork` was called. + +### Running simplecov against spawned subprocesses + +Perhaps you're testing a ruby script with `PTY.spawn` or `Open3.popen`, or `Process.spawn` or etc. +SimpleCov can cover this too. + +Add a .simplecov_spawn.rb file to your project root +```ruby +# .simplecov_spawn.rb +require 'simplecov' # this will also pick up whatever config is in .simplecov + # so ensure it just contains configuration, and doesn't call SimpleCov.start. +SimpleCov.command_name 'spawn' # As this is not for a test runner directly, script doesn't have a pre-defined base command_name +SimpleCov.at_fork.call(Process.pid) # Use the per-process setup described previously +SimpleCov.start # only now can we start. +``` +Then, instead of calling your script directly, like: +```ruby +PTY.spawn('my_script.rb') do # ... +``` +Use bin/ruby to require the new .simplecov_spawn file, then your script +```ruby +PTY.spawn('ruby -r./.simplecov_spawn my_script.rb') do # ... +``` + ## Running coverage only on demand The Ruby STDLIB Coverage library that SimpleCov builds upon is *very* fast (on a ~10 min Rails test suite, the speed diff --git a/features/config_enable_for_subprocesses.feature b/features/config_enable_for_subprocesses.feature new file mode 100644 index 00000000..5d78b6b5 --- /dev/null +++ b/features/config_enable_for_subprocesses.feature @@ -0,0 +1,37 @@ +@rspec @process_fork + +Feature: + Coverage should include code run by subprocesses + + Background: + Given I'm working on the project "subprocesses" + + Scenario: Coverage has seen the subprocess line + When I open the coverage report generated with `bundle exec rspec spec/simple_spec.rb` + Then I should see the groups: + | name | coverage | files | + | All Files | 100.0% | 1 | + + Scenario: The at_fork proc + Given a file named ".simplecov" with: + """ + SimpleCov.enable_for_subprocesses true + SimpleCov.command_name "parent process name" + SimpleCov.at_fork do |_pid| + SimpleCov.command_name "child process name" + SimpleCov.start + end + """ + When I open the coverage report generated with `bundle exec rspec spec/simple_spec.rb` + Then I should see the groups: + | name | coverage | files | + | All Files | 100.0% | 1 | + And the report should be based upon: + | child process name | + | parent process name | + + Scenario: The documentation on .simplecov_spawn + When I open the coverage report generated with `bundle exec rspec spec/spawn_spec.rb` + Then I should see the groups: + | name | coverage | files | + | All Files | 100.0% | 1 | diff --git a/features/support/env.rb b/features/support/env.rb index 36279d3c..c59526dc 100644 --- a/features/support/env.rb +++ b/features/support/env.rb @@ -47,9 +47,13 @@ end Before("@rails6") do - # Rails 6 only supports Ruby 2.5+ and amazingly because string comparison - # goes beginning to end the string comparison _should_ work - skip_this_scenario if RUBY_VERSION < "2.5" + # Rails 6 only supports Ruby 2.5+ + skip_this_scenario if Gem::Version.new(RUBY_VERSION) < Gem::Version.new("2.5") +end + +Before("@process_fork") do + # Process.fork is NotImplementedError in jruby + skip_this_scenario if defined?(RUBY_ENGINE) && RUBY_ENGINE == "jruby" end Aruba.configure do |config| diff --git a/lib/simplecov.rb b/lib/simplecov.rb index 32bdab3d..a278134f 100644 --- a/lib/simplecov.rb +++ b/lib/simplecov.rb @@ -52,6 +52,8 @@ class << self def start(profile = nil, &block) require "coverage" initial_setup(profile, &block) + require_relative "./simplecov/process" if SimpleCov.enabled_for_subprocesses? + @result = nil self.pid = Process.pid diff --git a/lib/simplecov/configuration.rb b/lib/simplecov/configuration.rb index bba523a5..1b881c4d 100644 --- a/lib/simplecov/configuration.rb +++ b/lib/simplecov/configuration.rb @@ -196,6 +196,52 @@ def at_exit(&block) @at_exit ||= proc { SimpleCov.result.format! } end + # gets or sets the enabled_for_subprocess configuration + # when true, this will inject SimpleCov code into Process.fork + def enable_for_subprocesses(value = nil) + @enable_for_subprocesses = value unless value.nil? + @enable_for_subprocesses || false + end + + # gets the enabled_for_subprocess configuration + def enabled_for_subprocesses? + enable_for_subprocesses + end + + # + # Gets or sets the behavior to start a new forked Process. + # + # By default, it will add " (Process #{pid})" to the command_name, and start SimpleCov in quiet mode + # + # Configure with: + # + # SimpleCov.at_fork do |pid| + # SimpleCov.start do + # # This needs a unique name so it won't be ovewritten + # SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})" + # # be quiet, the parent process will be in charge of using the regular formatter and checking coverage totals + # SimpleCov.print_error_status = false + # SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter + # SimpleCov.minimum_coverage 0 + # # start + # SimpleCov.start + # end + # end + # + def at_fork(&block) + @at_fork = block if block_given? + @at_fork ||= lambda { |pid| + # This needs a unique name so it won't be ovewritten + SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})" + # be quiet, the parent process will be in charge of using the regular formatter and checking coverage totals + SimpleCov.print_error_status = false + SimpleCov.formatter SimpleCov::Formatter::SimpleFormatter + SimpleCov.minimum_coverage 0 + # start + SimpleCov.start + } + end + # # Returns the project name - currently assuming the last dirname in # the SimpleCov.root is this. diff --git a/lib/simplecov/process.rb b/lib/simplecov/process.rb new file mode 100644 index 00000000..fbe59068 --- /dev/null +++ b/lib/simplecov/process.rb @@ -0,0 +1,19 @@ +# frozen_string_literal: true + +module Process + class << self + def fork_with_simplecov(&block) + if defined?(SimpleCov) && SimpleCov.running + fork_without_simplecov do + SimpleCov.at_fork.call(Process.pid) + block.call if block_given? + end + else + fork_without_simplecov(&block) + end + end + + alias fork_without_simplecov fork + alias fork fork_with_simplecov + end +end diff --git a/test_projects/subprocesses/.simplecov b/test_projects/subprocesses/.simplecov new file mode 100644 index 00000000..4e3d9215 --- /dev/null +++ b/test_projects/subprocesses/.simplecov @@ -0,0 +1,5 @@ +SimpleCov.enable_for_subprocesses true +# different versions of ruby were tracking different numbers of files. idk why. +# lets only worry about one file. +SimpleCov.add_filter /command/ +SimpleCov.add_filter /spawn/ diff --git a/test_projects/subprocesses/.simplecov_spawn.rb b/test_projects/subprocesses/.simplecov_spawn.rb new file mode 100644 index 00000000..14c9b5af --- /dev/null +++ b/test_projects/subprocesses/.simplecov_spawn.rb @@ -0,0 +1,5 @@ +require 'bundler/setup' +require 'simplecov' +SimpleCov.command_name 'spawn' +SimpleCov.at_fork.call(Process.pid) +SimpleCov.start diff --git a/test_projects/subprocesses/lib/command b/test_projects/subprocesses/lib/command new file mode 100755 index 00000000..11091537 --- /dev/null +++ b/test_projects/subprocesses/lib/command @@ -0,0 +1,7 @@ +#!/usr/bin/env ruby + +require_relative './subprocesses' + +Subprocesses.new.run + +puts 'done' diff --git a/test_projects/subprocesses/lib/subprocesses.rb b/test_projects/subprocesses/lib/subprocesses.rb new file mode 100644 index 00000000..877d4c56 --- /dev/null +++ b/test_projects/subprocesses/lib/subprocesses.rb @@ -0,0 +1,21 @@ +class Subprocesses + def run + method_called_in_parent_process + + pid = Process.fork do + method_called_by_subprocess + end + + Process.wait(pid) + + true + end + + def method_called_in_parent_process + true + end + + def method_called_by_subprocess + true + end +end diff --git a/test_projects/subprocesses/spec/simple_spec.rb b/test_projects/subprocesses/spec/simple_spec.rb new file mode 100644 index 00000000..e289817a --- /dev/null +++ b/test_projects/subprocesses/spec/simple_spec.rb @@ -0,0 +1,7 @@ +require_relative "spec_helper" + +describe Subprocesses do + it "call things" do + expect(subject.run).to be true + end +end diff --git a/test_projects/subprocesses/spec/spawn_spec.rb b/test_projects/subprocesses/spec/spawn_spec.rb new file mode 100644 index 00000000..37824edc --- /dev/null +++ b/test_projects/subprocesses/spec/spawn_spec.rb @@ -0,0 +1,10 @@ +require 'spec_helper' +require 'open3' +describe 'spawn' do + it 'calls things' do + Dir.chdir(File.expand_path('..', __dir__)) do + stdout, exitstatus = Open3.capture2("ruby -r./.simplecov_spawn lib/command") + expect(stdout.chomp).to eq 'done' + end + end +end diff --git a/test_projects/subprocesses/spec/spec_helper.rb b/test_projects/subprocesses/spec/spec_helper.rb new file mode 100644 index 00000000..34172054 --- /dev/null +++ b/test_projects/subprocesses/spec/spec_helper.rb @@ -0,0 +1,6 @@ +# frozen_string_literal: true + +require "simplecov" +SimpleCov.start + +require_relative "../lib/subprocesses.rb"