Using Cucumber CLI with "--dry-run" and "--format <message|html>" throws NilClass error

[cbochs]

Describe the bug

It seems that formatting is broken for message and html when dry-running features. This might be a duplicate of #1488? However, I thought it might be easier to identify with the reproduction steps (listed below). I have tested this on multiple Cucumber versions (4.1.0 up to 5.2.0) and all seem to demonstrate the same behaviour.

Notes:

• The bug doesn't occur when there are no feature files. It will only show up when there are 1 or more feature files available.
• The bug occurs with --format html or --format message. However, most other format flags work just fine (e.g. pretty, json, summary).
• This is probably obvious, but this bug halts any further output, causing the program to exit (exit code 2)

To Reproduce

1. Create simple Cucumber project cucumber --init
2. Add basic feature file + step definitions

   # features/test.feature
   Feature Test Feature
     Scenario: Test Scenario
       Given I am a test
       Then  I should succeed
   
   # features/step_definitions/test.rb
   Given(/^I am a test$/) do
     true
   end
   
   Then(/^I should succeed$/) do
     true
   end

3. Run cucumber --dry-run with --format html or --format message
4. See following error:

   {"meta":{"protocolVersion":"13.2.1","implementation":{"name":"cucumber-ruby","version":"5.2.0"},"runtime":{"name":"ruby","version":"2.6.5"},"os":{"name":"darwin18","version":"Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64"},"cpu":{"name":"x86_64"}}}
   {"stepDefinition":{"id":"3cef57c7-5c0c-40b9-890f-aff05eb5acac","pattern":{"source":"^I am a test$","type":"REGULAR_EXPRESSION"},"sourceReference":{"uri":"features/step_definitions/test.rb","location":{"line":1}}}}
   {"stepDefinition":{"id":"52407b12-043d-4731-985d-1dbfba1daf77","pattern":{"source":"^I give no shits$","type":"REGULAR_EXPRESSION"},"sourceReference":{"uri":"features/step_definitions/test.rb","location":{"line":5}}}}
   {"stepDefinition":{"id":"341cdf1f-7a46-4e3c-b919-893dc6e426b4","pattern":{"source":"^I should succeed$","type":"REGULAR_EXPRESSION"},"sourceReference":{"uri":"features/step_definitions/test.rb","location":{"line":9}}}}
   {"source":{"uri":"features/test.feature","data":"Feature: Test Feature\n  Test Feature description\n\n  Scenario: Test Scenario\n    Given I am a test\n    And   I give no shits\n    Then  I should succeed\n","mediaType":"text/x.cucumber.gherkin+plain"}}
   {"gherkinDocument":{"feature":{"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"Test Feature","description":"  Test Feature description","children":[{"scenario":{"id":"4a43760a-c976-4337-aad7-fbb82150d8b1","location":{"line":4,"column":3},"keyword":"Scenario","name":"Test Scenario","steps":[{"location":{"line":5,"column":5},"keyword":"Given ","text":"I am a test","id":"7033d3ca-118a-40db-b9b1-d5535e0aade5"},{"location":{"line":6,"column":5},"keyword":"And ","text":"I give no shits","id":"c8e63287-a57d-4e72-83fb-5bac91779014"},{"location":{"line":7,"column":5},"keyword":"Then ","text":"I should succeed","id":"4d1e847b-241c-4127-81c8-028f4bbb8d59"}]}}]},"uri":"features/test.feature"}}
   {"pickle":{"uri":"features/test.feature","id":"907b0303-536b-488a-9e65-655706bbca57","name":"Test Scenario","language":"en","astNodeIds":["4a43760a-c976-4337-aad7-fbb82150d8b1"],"steps":[{"id":"51823da3-c98b-4999-8b75-d1068cc47fb5","astNodeIds":["7033d3ca-118a-40db-b9b1-d5535e0aade5"],"text":"I am a test"},{"id":"7d76527f-8686-4984-9872-412f786b94f0","astNodeIds":["c8e63287-a57d-4e72-83fb-5bac91779014"],"text":"I give no shits"},{"id":"970177cf-9378-47be-a428-3bca15fa5bbd","astNodeIds":["4d1e847b-241c-4127-81c8-028f4bbb8d59"],"text":"I should succeed"}]}}
   {"testCaseStarted":{"id":"f5c5284d-3686-4105-aab0-db1807001a18","testCaseId":"f8b23876-3b51-448d-b9ac-fcac2bd96878","timestamp":{"seconds":"1610335526","nanos":842180000},"attempt":1}}
   undefined method `id' for nil:NilClass (NoMethodError)
   .../gems/cucumber-5.2.0/lib/cucumber/formatter/query/test_case_started_by_test_case.rb:22:in `test_case_started_id_by_test_case'
   .../gems/cucumber-5.2.0/lib/cucumber/formatter/message_builder.rb:251:in `test_case_started_id'
   .../gems/cucumber-5.2.0/lib/cucumber/formatter/message_builder.rb:177:in `on_test_step_started'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/event_bus.rb:34:in `block in broadcast'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/event_bus.rb:34:in `each'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/event_bus.rb:34:in `broadcast'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/event_bus.rb:40:in `method_missing'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/runner.rb:28:in `test_step'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/step.rb:23:in `describe_to'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:31:in `block (3 levels) in describe_to'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:30:in `each'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:30:in `block (2 levels) in describe_to'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:81:in `compose_around_hooks'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:29:in `block in describe_to'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/runner.rb:19:in `test_case'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:28:in `describe_to'
   .../gems/cucumber-5.2.0/lib/cucumber/filters/activate_steps.rb:12:in `test_case'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/case.rb:28:in `describe_to'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/filters/locations_filter.rb:18:in `block in done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/filters/locations_filter.rb:17:in `each'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/filters/locations_filter.rb:17:in `done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/filter.rb:62:in `done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/compiler.rb:31:in `done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core/gherkin/parser.rb:46:in `done'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core.rb:35:in `parse'
   .../gems/cucumber-core-8.0.1/lib/cucumber/core.rb:24:in `compile'
   .../gems/cucumber-5.2.0/lib/cucumber/runtime.rb:79:in `run!'
   .../gems/cucumber-5.2.0/lib/cucumber/cli/main.rb:29:in `execute!'
   .../gems/cucumber-5.2.0/bin/cucumber:9:in `<top (required)>'
   /usr/local/bin/cucumber:23:in `load'
   /usr/local/bin/cucumber:23:in `<main>'
   /usr/local/bin/ruby_executable_hooks:24:in `eval'
   /usr/local/bin/ruby_executable_hooks:24:in `<main>'
   

Expected behavior

Full output without above error.

Context & Motivation

I am attempting to get all available Cucumber tests. Typically I would do this using --format json, but it seems that formatter is being deprecated so I opted to try out --format message (which causes the above error). If there is a better way to list all cucumber tests that someone knows of, please let me know. Thanks!

Screenshots

N/A

Your Environment

• Cucumber: 5.2.0 (tested on versions 4.1.0 up to 5.2.0 as well, none worked)
• Ruby: 2.6.5
• OS: macOS Catalina

Additional context

N/A

[cbochs]

After a little bit of digging, I found that @test_case_by_step_id is not populated in the message_builder.rb before the on_test_step_started method is called, causing a lookup to return a nil value. This is because the event BroadcastTestCaseReadyEvent (subscribed to by the message_builder here) is not triggered.

A little more digging shows that the BroadcastTestCaseReadyEvent is purposefully from the filters when the --dry-run flag is set (see: runtime.rb#L243). This change occurred during the initial PR to add the message formatter: #1387. I'm not very familiar with the cucumber-ruby codebase, so @vincent-psarga Is there a specific reason this was omitted from dry runs?