Using parallel_split_test, seeing Errno::ENOENT: No such file or directory - /usr/sbin/sysctl

[pamgluss-slack]

Error we're getting:
bundle exec parallel_split_test spec --tag ~unstable --tag ~skip_regression 17:29:50 bundler: failed to load command: parallel_split_test (/mnt/jenkinsslave/workspace/webapp-regression-api/tests/rspec-api-tests/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bin/parallel_split_test) 17:29:50 Errno::ENOENT: No such file or directory - /usr/sbin/sysctl

The time we started seeing this was 3:30 PM on our Jenkins builds. This lines up with the release of 1.20.2. A workaround is for us to use 1.20.1 but I wanted to bring this to yalls attention.

Ruby version 2.4.1
Rspec Version 3.9.0

[grosser]

thx
comes from

parallel/lib/parallel/processor_count.rb

Line 16 in f456129

IO.popen("/usr/sbin/sysctl -n hw.physicalcpu").read.to_i

are you running on mac ?
what does /usr/sbin/sysctl -n hw.physicalcpu return ?
what is which sysctl ?

[grosser]

/cc @pedromartinez

[pedromartinez]

pedro@pedros-mac-mini ~ % /usr/sbin/sysctl -n hw.physicalcpu
6

[pedromartinez]

I'll open a PR to revert.

[pedromartinez]

#301

[pedromartinez]

Note:

physical_processor_count appears to be used in the parallel_split_test gem but not in parallel_tests.

Even after my PR, parallel tests still returned 12 for processor_count.

Reference:
https://github.com/grosser/parallel_split_test/blob/9ea3365898311165da5dd426f3b5ff4dc71fc515/lib/parallel_split_test.rb#L19

[grosser]

@pamgluss-slack can you try and find out why sysctl is missing for you ?

[grosser]

it uses processor_count https://github.com/grosser/parallel_tests/blob/master/lib/parallel_tests.rb#L20
that seems bad since either one or the other is better for tests, so if you can find out if 6/12 is best then we can change one or the other

[grosser]

let's revert for now since it looks like we need a different fix

[pamgluss-slack]

I can get details for you guys in the morning PST, but I believe Jenkins runs Linux machines

os.arch | amd64
os.name | Linux
os.version | 4.4.0-1106-aws

[pedromartinez]

@grosser I agree, it should probably use the same method for both libraries.

Here are two different work projects.

A smaller project:

6x:

426 examples, 0 failures
Took 50 seconds

12x:

426 examples, 0 failures
Took 59 seconds

and

A larger project:

6x:

8949 examples, 0 failures, 4 pendings
Took 406 seconds (6:46)

12x:

8949 examples, 0 failures, 4 pendings
Took 332 seconds (5:32)

[pedromartinez]

For reference:

pedro@pedros-mac-mini % sysctl -n hw.physicalcpu
6

pedro@pedros-mac-mini  % sysctl -n hw.ncpu       
12

[grosser]

seems to be unclear, so I'd rather use 12 since changing parallel_tests impacts more peoples workflow

…

On Wed, Aug 25, 2021 at 7:44 AM Pedro Martinez ***@***.***> wrote: @grosser <https://github.com/grosser> I agree, it should probably use the same method for both libraries. Here are two different work projects. A smaller project: 6x: 426 examples, 0 failures Took 50 seconds 12x: 426 examples, 0 failures Took 59 seconds and A larger project: 6x: 8949 examples, 0 failures, 4 pendings Took 406 seconds (6:46) 12x: 8949 examples, 0 failures, 4 pendings Took 332 seconds (5:32) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACYZ4BNCV7ASWLC376HNTT6T6URANCNFSM5CX6GSNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[pedromartinez]

Agreed!

[grosser]

done

[grosser]

should be fixed in 1.21.0