Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Skip TestIntegrationSingle#test_closed_listener on macOS #2738

Closed
wants to merge 1 commit into from

Conversation

dentarg
Copy link
Member

@dentarg dentarg commented Oct 30, 2021

In an effort to have green CI more often.

It was introduced in #2700

I've seen it fail like this on macos-10.15 and Ruby

      1) Error:
    TestIntegrationSingle#test_closed_listener:
    EOFError: end of file reached
        /Users/runner/work/puma/puma/test/helpers/integration.rb:160:in `readpartial'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:160:in `block (2 levels) in read_body'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:159:in `loop'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:159:in `block in read_body'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:158:in `read_body'
        /Users/runner/work/puma/puma/test/test_integration_single.rb:187:in `test_closed_listener'
        /Users/runner/work/puma/puma/test/helper.rb:82:in `block (4 levels) in run'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:98:in `call'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:98:in `timeout'
        /Users/runner/work/puma/puma/test/helper.rb:80:in `block (3 levels) in run'

And like this on

      1) Error:
    TestIntegrationSingle#test_closed_listener:
    TimeoutEveryTestCase::TestTookTooLong: execution expired
        /Users/runner/work/puma/puma/test/helpers/integration.rb:124:in `gets'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:124:in `connect'
        /Users/runner/work/puma/puma/test/test_integration_single.rb:187:in `test_closed_listener'
        /Users/runner/work/puma/puma/test/helper.rb:82:in `block (4 levels) in run'
        /Users/runner/hostedtoolcache/Ruby/2.4.10/x64/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
        /Users/runner/work/puma/puma/test/helper.rb:80:in `block (3 levels) in run'

Also: Can't see that Puma::IS_OSX has ever existed, so fix that (changed in #2576).

In an effort to have green CI more often.

It was introduced in puma#2700

I've seen it fail like this on macos-10.15 and Ruby
- 2.2: https://github.com/puma/puma/runs/4055009489?check_suite_focus=true#step:9:739
- 2.3: https://github.com/puma/puma/runs/4055009508?check_suite_focus=true#step:9:661
- macos-11 2.4: https://github.com/puma/puma/runs/4055009711?check_suite_focus=true#step:9:667

      1) Error:
    TestIntegrationSingle#test_closed_listener:
    EOFError: end of file reached
        /Users/runner/work/puma/puma/test/helpers/integration.rb:160:in `readpartial'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:160:in `block (2 levels) in read_body'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:159:in `loop'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:159:in `block in read_body'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:32:in `catch'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:158:in `read_body'
        /Users/runner/work/puma/puma/test/test_integration_single.rb:187:in `test_closed_listener'
        /Users/runner/work/puma/puma/test/helper.rb:82:in `block (4 levels) in run'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:98:in `call'
        /Users/runner/hostedtoolcache/Ruby/2.2.10/x64/lib/ruby/2.2.0/timeout.rb:98:in `timeout'
        /Users/runner/work/puma/puma/test/helper.rb:80:in `block (3 levels) in run'

And like this on
- macos-10.15 Ruby 2.4: https://github.com/puma/puma/runs/4055009526?check_suite_focus=true#step:9:618
- macos-11 Ruby 2.3: https://github.com/MSP-Greg/puma/runs/3616720219?check_suite_focus=true#step:9:574 (when it was introduced)

      1) Error:
    TestIntegrationSingle#test_closed_listener:
    TimeoutEveryTestCase::TestTookTooLong: execution expired
        /Users/runner/work/puma/puma/test/helpers/integration.rb:124:in `gets'
        /Users/runner/work/puma/puma/test/helpers/integration.rb:124:in `connect'
        /Users/runner/work/puma/puma/test/test_integration_single.rb:187:in `test_closed_listener'
        /Users/runner/work/puma/puma/test/helper.rb:82:in `block (4 levels) in run'
        /Users/runner/hostedtoolcache/Ruby/2.4.10/x64/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
        /Users/runner/work/puma/puma/test/helper.rb:80:in `block (3 levels) in run'

Can't see that `Puma::IS_OSX` has ever existed, so fix that (changed in puma#2576).
@dentarg dentarg requested a review from MSP-Greg October 30, 2021 20:07
@MSP-Greg
Copy link
Member

@dentarg

Sorry for the error with Puma::IS_OSX. I was porting code from the 'new test framework' (which I really need to get back to working on). One issue is the TestIntegration#connect method writes and reads the client socket, so it's messy to separate read vs write errors. The new test framework split that...

Anyway, can you have look at the first two commits in https://github.com/MSP-Greg/puma/commits/fix-ci? Ran it twice in CI, didn't see the failure addressed here, but there's more.

I've noticed with some cron CI I have that Actions seems to be running slower. I think all platforms share 'runner' code, and maybe it's 'interrupting' the 'code under test'. Some of the IO.open IO's are having errors on read, which I don't recall seeing before. I'm going to see if adding a delay for darwin helps...

@dentarg
Copy link
Member Author

dentarg commented Oct 30, 2021

@MSP-Greg Those commits look great, thanks for doing that, if open a PR we can merge them instead of this one

@dentarg
Copy link
Member Author

dentarg commented Oct 30, 2021

I think the performance of Actions runs can vary widely (VMs on shared hardware have that problem) and especially macOS seeing GitHub had a very slow onboarding of macOS 11, I suspect resources are limited in that area.

@MSP-Greg MSP-Greg mentioned this pull request Oct 31, 2021
7 tasks
@MSP-Greg
Copy link
Member

@dentarg See PR #2739

I think the performance of Actions runs can vary widely

Agreed. Lately, it seems like CI is also running slower...

@dentarg dentarg closed this Oct 31, 2021
@dentarg dentarg deleted the macos/skip/test_closed_listener branch October 31, 2021 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants