Inspec-1.51.* fail with --concurrency option in kitchen #167

Linutux42 · 2018-02-20T17:24:24Z

Hello,

I just found out a bug and I am not sure if the issue is within kitchen-inspec or inspec itself.
When I use --concurrency option while doing kitchen verify I get the following error:

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #verify action: [undefined method `each' for nil:NilClass] on test-cookbook-centos-7-x86-64
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

If I remove the --concurrency option the kitchen verify finishes without issue.
I got the same result on two dinstinct servers and even on my workstation.

Here are the version I tried:

kitchen-inspec: 0.22.0, 0.20.0
inspec: 1.51.*

It works with inspec-1.49.2 though.

I could not test it with inspec-2.* because I have the following issue, even though I have inspec-2.0.17 installed. Should I open another issue about that or is it an issue on my end only ?

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'inspec' verifier from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Let me know if you need any more informations

Thanks

The text was updated successfully, but these errors were encountered:

cheeseplus · 2018-02-22T18:26:47Z

The first error we would need some more context, namely the .kitchen.yml and the concurrency value passed (ideally the actual command). The latter failure means that whatever kitchen binary you are using can't see kitchen-inspec, this is probably a local environment condition - i.e. ChefDK not installed correctly or something not set correctly in the .kitchen.yml.

Linutux42 · 2018-02-23T15:16:47Z

Hey @cheeseplus,

1st error:

Any value superior to 1 ended up with this issue. I tried from 1 to 10.
Command :
$> kitchen verify --concurency 10

.kitchen.yml:

---
driver:
  name: vagrant
  customize:
    memory: 1024
    cpus: 1
    natnet1: "192.168/16"
  boot_timeout: 3600

transport:
  name: sftp

provisioner:
  name: chef_zero
  require_chef_omnibus: latest
  chef_omnibus_url: http://chef-omnibus
verifier:
  name: inspec

platforms:
- name: centos-6-x86_64
  driver:
    box: centos-6-x86_64
    box_url: http://repository/vagrant-boxes/centos-6-x86_64.box
    box_check_update: true

- name: centos-7-x86_64
  driver:
    box: centos-7-x86_64
    box_url: http://repository/vagrant-boxes/centos-7-x86_64.box
    box_check_update: true

suites:
- name: test_cookbook
  data_bags_path: ../data_bags/
  run_list:
    - recipe[test_cookbook::default]
  verifier:
    inspec_tests:
      - test/basic/default.rb

I'll check the second error later :)

Thank you,
Maxence

cheeseplus · 2018-02-23T15:34:30Z

Also need more of the log or preferably the debug log for the first error. I think the second error has more to do with installing the InSpec package (which is just inspec) vs installing via ChefDK (which doesn't have inspec 2 yet).

ghost · 2018-03-03T23:48:30Z

#119

gionn · 2018-05-04T08:37:48Z

Hitting on me as well.

Seems that we have a nice race condition here

E, [2018-05-04T10:04:24.224391 #25690] ERROR -- kong-ubuntu-1404: ------Exception-------
E, [2018-05-04T10:04:24.224454 #25690] ERROR -- kong-ubuntu-1404: Class: NoMethodError
E, [2018-05-04T10:04:24.224489 #25690] ERROR -- kong-ubuntu-1404: Message: undefined method `each' for nil:NilClass
E, [2018-05-04T10:04:24.224517 #25690] ERROR -- kong-ubuntu-1404: ----------------------
E, [2018-05-04T10:04:24.224538 #25690] ERROR -- kong-ubuntu-1404: ------Backtrace-------
E, [2018-05-04T10:04:24.224558 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/reporters/cli.rb:46:in `render'
E, [2018-05-04T10:04:24.224579 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/reporters.rb:24:in `render'
E, [2018-05-04T10:04:24.224619 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:111:in `block in render_output'
E, [2018-05-04T10:04:24.224649 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:110:in `each'
E, [2018-05-04T10:04:24.224683 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:110:in `render_output'
E, [2018-05-04T10:04:24.224716 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:134:in `run_tests'
E, [2018-05-04T10:04:24.224748 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/inspec-2.1.0/lib/inspec/runner.rb:104:in `run'
E, [2018-05-04T10:04:24.224778 #25690] ERROR -- kong-ubuntu-1404: /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/kitchen-inspec-0.23.1/lib/kitchen/verifier/inspec.rb:95:in `call'
E, [2018-05-04T10:04:24.224809 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:443:in `block in verify_action'
E, [2018-05-04T10:04:24.224839 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:536:in `synchronize_or_call'
E, [2018-05-04T10:04:24.224870 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:498:in `block in action'
E, [2018-05-04T10:04:24.224900 #25690] ERROR -- kong-ubuntu-1404: /opt/chefdk/embedded/lib/ruby/2.4.0/benchmark.rb:293:in `measure'
E, [2018-05-04T10:04:24.224930 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:497:in `action'
E, [2018-05-04T10:04:24.224961 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:435:in `verify_action'
E, [2018-05-04T10:04:24.224994 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:368:in `block in transition_to'
E, [2018-05-04T10:04:24.225027 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:367:in `each'
E, [2018-05-04T10:04:24.225058 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:367:in `transition_to'
E, [2018-05-04T10:04:24.225089 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/instance.rb:157:in `verify'
E, [2018-05-04T10:04:24.225122 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:197:in `public_send'
E, [2018-05-04T10:04:24.225155 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:197:in `run_action_in_thread'
E, [2018-05-04T10:04:24.225186 #25690] ERROR -- kong-ubuntu-1404: /home/gionn/.chefdk/gem/ruby/2.4.0/gems/test-kitchen-1.21.1/lib/kitchen/command.rb:169:in `block (2 levels) in run_action'
E, [2018-05-04T10:04:24.225216 #25690] ERROR -- kong-ubuntu-1404: ----End Backtrace-----

jippi · 2018-06-18T14:39:04Z

I'm seeing this as well in our builds using inspec (1.51.21), kitchen-inspec (0.23.1) and chef (13.8.5)

bobchaos · 2018-09-24T20:13:03Z

Issue persists with the following:
chefdk 3.2.30
inspec 2.2.70
kitchen 1.23.2
Fixing this would speed up my CI pipeline significantly, let me know if I can provide additional debug data or anything such.

gionn · 2018-09-25T09:29:44Z

Workaround with GNU parallel:

apt install -y parallel
kl=$(kitchen list -b -l fatal)
parallel --no-notice -j8 --tag kitchen test -d always --no-color --no-log-overwrite {} ::: $kl

chris-rock · 2018-09-25T10:31:11Z

Its a bit tricky to implement parallel execution. The challenge is not so much in inspec itself, the challenge is that we use RSpec undercover and its creating a global state. I see two doable options:

instead of using the inspec's ruby classes, we exec to the inspec bin with the arguments
we install inspec on each machine and (not preferred, since its way slower)

Essentially, we need to make sure that the execution context is not shared

@gionn has an excellent workaround. Thank you for sharing!

inspec/kitchen-inspec#167 leads to failures when running tests in parallel.

tas50 · 2019-08-20T20:31:35Z

See #119 for more context

clintoncwolfe · 2020-09-02T17:38:25Z

This has been re-created with the kitchen-vcenter driver as part of the testing for chef/chef-workstation#1442 and test-kitchen/test-kitchen#1677. TK 1677 allows plugins to declare that they are not thread safe - which as was mentioned is not possible for InSpec due to RSpec being non-thread safe - see RSpec discussion.

TK 1678 enables this plugin to declare that is not thread safe and the Verify stage should be run serially, which in my testing has been stable and a negligible slowdown under vcenter, vagrant, and docken.

stuart12 added a commit to criteo-cookbooks/mapr that referenced this issue Feb 18, 2019

Disable concurrency on travis CI tests due to bug

de2462c

inspec/kitchen-inspec#167 leads to failures when running tests in parallel.

stuart12 mentioned this issue Feb 18, 2019

Disable concurrency on travis CI tests due to bug criteo-cookbooks/mapr#13

Merged

stevenoneill mentioned this issue Jun 28, 2019

Test Kitchen Concurrency Fails test-kitchen/test-kitchen#1568

Closed

zenspider added Aspect: Correctness Priority: Medium Type: Bug Doesn't work as expected. labels Sep 17, 2019

tas50 removed the Aspect: Correctness label Jun 4, 2020

This was referenced Sep 2, 2020

[test-kitchen] Toggle concurrency per-plugin chef/chef-workstation#1442

Closed

Declare kitchen-inspec to be run serially #259

Merged

Random failures when using -c with kitchen verify test-kitchen/test-kitchen#1530

Closed

tas50 closed this as completed in #259 Sep 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inspec-1.51.* fail with --concurrency option in kitchen #167

Inspec-1.51.* fail with --concurrency option in kitchen #167

Linutux42 commented Feb 20, 2018 •

edited

cheeseplus commented Feb 22, 2018

Linutux42 commented Feb 23, 2018 •

edited

cheeseplus commented Feb 23, 2018

ghost commented Mar 3, 2018

gionn commented May 4, 2018

jippi commented Jun 18, 2018 •

edited

bobchaos commented Sep 24, 2018

gionn commented Sep 25, 2018

chris-rock commented Sep 25, 2018 •

edited

tas50 commented Aug 20, 2019

clintoncwolfe commented Sep 2, 2020

Inspec-1.51.* fail with --concurrency option in kitchen #167

Inspec-1.51.* fail with --concurrency option in kitchen #167

Comments

Linutux42 commented Feb 20, 2018 • edited

cheeseplus commented Feb 22, 2018

Linutux42 commented Feb 23, 2018 • edited

cheeseplus commented Feb 23, 2018

ghost commented Mar 3, 2018

gionn commented May 4, 2018

jippi commented Jun 18, 2018 • edited

bobchaos commented Sep 24, 2018

gionn commented Sep 25, 2018

chris-rock commented Sep 25, 2018 • edited

tas50 commented Aug 20, 2019

clintoncwolfe commented Sep 2, 2020

Linutux42 commented Feb 20, 2018 •

edited

Linutux42 commented Feb 23, 2018 •

edited

jippi commented Jun 18, 2018 •

edited

chris-rock commented Sep 25, 2018 •

edited