Capybara hangs with multiple drivers if there are not enough threads

[trcarden]

Meta

Capybara Version: 3.25.0
Driver Information Latest ChromeDriver on Chrome 75
Puma: 4.0.0

Expected Behavior

I can run the test suite successfully

Actual Behavior

The test suite errors out with Net::ReadTimeout with #<TCPSocket:(closed)> or sometimes Net::OpenTimeout

  # [Redacted]/gems/webmock-3.5.1/lib/webmock/http_lib_adapters/net_http.rb:136:in `start_with_connect_without_finish'
  # [Redacted]/gems/webmock-3.5.1/lib/webmock/http_lib_adapters/net_http.rb:104:in `request'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server.rb:54:in `block in responsive?'
  # [Redacted]/gems/webmock-3.5.1/lib/webmock/http_lib_adapters/net_http.rb:123:in `start_without_connect'
  # [Redacted]/gems/webmock-3.5.1/lib/webmock/http_lib_adapters/net_http.rb:150:in `start'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server/checker.rb:36:in `make_request'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server/checker.rb:32:in `https_request'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server/checker.rb:16:in `rescue in request'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server/checker.rb:13:in `request'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server.rb:54:in `responsive?'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/server.rb:71:in `boot'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/session.rb:91:in `initialize'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara.rb:415:in `new'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara.rb:415:in `block in session_pool'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara.rb:312:in `current_session'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/dsl.rb:46:in `page'
  # [Redacted]/gems/capybara-3.25.0/lib/capybara/dsl.rb:51:in `block (2 levels) in <module:DSL>'
  # ./spec/system/store/widget_spec.rb:21:in `block (3 levels) in <main>'
  # ./spec/rails_helper.rb:254:in `block (2 levels) in <top (required)>'
  # ------------------
  # --- Caused by: ---
  # Net::ReadTimeout:
  #   Net::ReadTimeout with #<TCPSocket:(closed)>
  #   [Redacted]/gems/webmock-3.5.1/lib/webmock/http_lib_adapters/net_http.rb:284:in `rbuf_fill'

Steps to reproduce

1. Create a test suite with 3 or more drivers (say one for chrome with iOS emulation, one with Android emulation and one for desktop -- you can use any drivers you want but just need to be 3 different registered drivers)

2. Enable SSL with Capybara using technique found here (Support configuring the puma server to use SSL #2028).

3. Run suite and wait for the timeouts

Example 3 drivers

Capybara.register_driver :chrome do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument('--window-size=1400,1400')
  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

Capybara.register_driver :chrome2 do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument('--window-size=1920,1080')
  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

Capybara.register_driver :chrome3 do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_emulation(device_name: 'Pixel 2')
  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

Example suite

describe 'Thing' do
  context 'chrome' do
    before do
      Capybara.current_driver = :chrome
      visit root_path
    end

    it 'truth' do
      expect(true).to be_truthy
    end
  end

  context 'chrome2' do
    before do
      Capybara.current_driver = :chrome2
      visit root_path
    end

    it 'truth' do
      expect(true).to be_truthy
    end
  end

  context 'chrome3' do
    before do
      Capybara.current_driver = :chrome3
      visit root_path
    end

    it 'truth' do
      expect(true).to be_truthy
    end
  end

end

Leads

I noticed when we boot a capybara session using SSL you get this:

Error in reactor loop escaped: System error: Undefined error: 0 - 0 (Puma::MiniSSL::SSLError)
[Redacted]/gems/puma-4.0.0/lib/puma/minissl.rb:43:in `read'
[Redacted]/gems/puma-4.0.0/lib/puma/minissl.rb:43:in `engine_read_all'
[Redacted]/gems/puma-4.0.0/lib/puma/minissl.rb:54:in `read_nonblock'
[Redacted]/gems/puma-4.0.0/lib/puma/minissl.rb:129:in `read_and_drop'
[Redacted]/gems/puma-4.0.0/lib/puma/minissl.rb:146:in `close'
[Redacted]/gems/puma-4.0.0/lib/puma/client.rb:138:in `close'
[Redacted]/gems/puma-4.0.0/lib/puma/reactor.rb:265:in `rescue in block in run_internal'
[Redacted]/gems/puma-4.0.0/lib/puma/reactor.rb:218:in `block in run_internal'
[Redacted]/gems/puma-4.0.0/lib/puma/reactor.rb:157:in `each'
[Redacted]/gems/puma-4.0.0/lib/puma/reactor.rb:157:in `run_internal'
[Redacted]/gems/puma-4.0.0/lib/puma/reactor.rb:311:in `block in run_in_thread'

I also notice from time to time that Puma keeps connections open for a long time and prevents terminating the server even if the webpage is fully loaded.

I also noticed if you revert back to just HTTP this doesn't happen (at least not at 3 drivers, maybe at a higher level).

Taken together my hypothesis was that we were using up all the threads on the server (either with stale connections, or not having enough connections to begin with for the number of drivers specified for usage -- especially if we make a HTTP request, fail and then make a HTTPS request like it looks like we are doing).

Workaround

Option 1

config.after :each do
  Capybara.current_session.driver.quit
end

If I shutdown the browsers after each test everything seems to work. Which lead me to believe that the server's default 4 threads might be an issue

Option 2

Capybara.configure do |config|
  key_file = Rails.root.join('config', 'certs', 'insecure.key')
  cert_file = Rails.root.join('config', 'certs', 'insecure.crt')
  config.server = :puma, {
    Host: "ssl://#{Capybara.server_host}?key=#{key_file}&cert=#{cert_file}",
    Threads: '0:10'
  }
end

Increasing the max thread count also seemed to resolve the issue.

Idea

Maybe some combo of increasing the default number of threads and shutting down drivers if there are more than a few open would help resolve the issue.

[twalpole]

I'm not sure there's anything Capybara should do about this, since it seems to purely be a user configuration issue. Capybara doesn't control the number of opened requests, that's really dependent on your app and how many simultaneous requests it opens, and Capybara automatically shutting down drivers by default is a non-starter from a performance perspective (and from a user expectation perspective). If the browser/puma are keeping open connections after the browser has been reset to about:blank that would seem to be a browser or puma issue. Additionally, only a user can know what their setup is going to need from a concurrency perspective. A README PR would be accepted if you can summarize the issue in a paragraph or so (probably in the Gotchas section)

[twalpole]

Note: another potential workaround would probably be to set Capybara.reuse_server = false to force Capybara to start a new server/app instance for each session rather than sharing one between all the sessions (would be conflicts if you're not using transactional tests though).

[twalpole]

This very much appears to be a bug or design flaw in puma - The fact that a persistent connection ties up a thread on the chance a request might come over that connection seems like not great behavior. This would really only be an issue when puma is run with no workers (which wouldn't be done in production) but it still seems a little nuts.

[twalpole]

Ok - setting queue_requests to false fixes the behavior as can be seen in the gist - https://gist.github.com/twalpole/52e4f833572086bee9900e09889852be
Setting it to true will cause the hanging behavior , when false everything works with only 1 thread. I'm going to change the default :puma config in Capybara to specify queue_requests: false

[twalpole]

Fixed via 5752b7e

[trcarden]

@twalpole nice find!

[benht-nps]

Hello,

After a little digging on upgrading I found this fix breaks our cucumber tests marked @javascript. Results in "This site cannot be reached"/"The connection was reset" for Chrome/Firefox. (Sorry didn't see any more useful information.)

I can't see a way to pass in/override the queue_requests option - is that currently possible?

[twalpole]

@benht-nps Interesting - I'm guessing your app is making more than 4 simultaneous requests - You can pass any options to puma via the server setting

Capybara.server = :puma, { queue_requests: true }

[benht-nps]

Thanks :) That fixes it.