Test system update

[MSP-Greg]

Description

1. Currently, all tests are wrapped by ::Timeout.timeout. Switched to using a constant defined in each test class. This PR changes so that only test classes that use some form of bundle install are using the timeout.

2. Created test/helpers/puma_socket.rb, which allows removal of duplicated code in several test files. This will also make it easier to re-arrange and/or split test files into more (smaller) files.

3. Made minor updates to some library files, most of which related to error handling.

4. Currently, some of the tests fail if a client socket raises Errno::ECONNRESET. I've got some code that queries the GitHub API with Net::HTTP, and it threw the same error, as GitHub has had network issues. Using the same URI, I tried several browsers, and they all retry on Errno::ECONNRESET. Changed test code to allow a small percentage of Errno::ECONNRESET errors.

5. CI is often running a full allocation of macos jobs at the end. Removed a few macos jobs to quicken the test time. This may be needed again, as GitHub will be supporting macos-13 and also arm64 macos.

Creating as a draft, not sure whether to break it up into lib file changes and test file changes, etc...

Your checklist for this pull request

• I have reviewed the guidelines for contributing to this repository.
• I have added (or updated) appropriate tests if this PR fixes a bug or adds a feature.
• My pull request is 100 lines added/removed or less so that it can be easily reviewed.
• If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
• If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
• I have updated the documentation accordingly.
• All new and existing tests passed, including Rubocop.

[MSP-Greg]

Fixed up test_out_of_band_server.rb, as a JRuby test froze in first CI run.

[dentarg]

Creating as a draft, not sure whether to break it up into lib file changes and test file changes, etc...

Yes, please break it up, I think this is a good tell 😄

Going by your list this is probably at least five different PRs?

[dentarg]

Remove this line? Or was it a mistake to comment this out?

[MSP-Greg]

Not a mistake. Can't quite remember the particulars, but localhost_authority will fail if some conditions aren't met. There's no reason to call it if the 'binds' do not include an ssl connection. I commented it out because I wanted to run the test suite several times, and forgot to remove the line.

[dentarg]

Got it. Would be nice to know what conditions.

[MSP-Greg]

It may have happened when running a single file locally, can't recall.

[dentarg]

GitHub hides the changes to test/helpers/integration.rb by default when viewing the diff so it is easy to miss those changes when reviewing.

Not to take anything away from the work done here, it is super great stuff @MSP-Greg 💪 but I do think we should aim for small specific PRs for changes to Puma

[MSP-Greg]

we should aim for small specific PRs for changes to Puma

I would generally agree. Unfortunately, the test suite is cluster-fu** of repetitive code, intermittent test failures, impossible to break apart into smaller files without more repetition, etc.

I think most of the changes are in the commits c521c5c, 1dadb22 and 2d4ab6c.

One is the changes to helper files and the addition of test/helpers/puma_socket.rb, and the two others update various test files to work with them.

Or, those three commits are tightly bound and need to be committed together. Normally I prefer to see PR's squashed, but maybe these should be left as mutiple commits.

I'll see if I can pull the library file commits from this into another PR.

[MSP-Greg]

General Concept

I started with two goals, first, make the test system more stable, and second, move towards more shared helper methods.

So, I shut off the Timeout thread wrapping each test, shut off the retry, and started running the test suite locally.

One main problem is running tests parallel. These run in threads. If any tests have blocking code, the tests slow down and may fail.

This means that client sockets should always use wait_readable, and read_nonblock would be preferred. Code should also close client sockets. Because of this, we should discourage use of *Socket.new in test code, and provide helper methods. Obviously, there will be exceptions.

Most tests either start a Puma server 'in-process' or spawn a server. In the future, having three helper classes, something like ServerInProc and ServerSpawn, along with a companion socket class would be a good setup. Implementing these classes will make it much easier to re-arrange tests. An example might be test_puma_server.rb, which contains 94 tests.

The current 'socket' helper in this PR only works with TCP & UNIX sockets. SSL sockets need to be added, and integrated with the inproc and spawn server helpers (inproc is yet to be written).

[MSP-Greg]

I've pulled all the code in lib files and Actions workflow files, so this now just contains test updates. I'll see whether it can be divided into smaller PR's, but it doesn't need to be squashed.

Also, as the goal was to make CI more reliable and also contain less repetitive code, it was a matter of converging toward a solution, so reversing that and splitting it up may be difficult. Some of the work involved small changes to remove blocking code.