Merge branch 'master' into json-require

.github/pull_request_template.md

@@ -5,7 +5,7 @@ Please describe your pull request. Thank you for contributing! You're the best.
 <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
 - [ ] I have reviewed the [guidelines for contributing](../blob/master/CONTRIBUTING.md) to this repository.
-- [ ] I have added an entry to [History.md](../blob/master/History.md) if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added `[changelog skip]` the pull request title.
+- [ ] I have added an entry to [History.md](../blob/master/History.md) if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added `[changelog skip]` or `[ci skip]` to the pull request title.
 - [ ] I have added appropriate tests if this PR fixes a bug or adds a feature.
 - [ ] My pull request is 100 lines added/removed or less so that it can be easily reviewed.
 - [ ] If this PR doesn't need tests (docs change), I added `[ci skip]` to the title of the PR.

.github/workflows/puma.yml

@@ -37,7 +37,6 @@ jobs:
         uses: MSP-Greg/setup-ruby-pkgs@v1
         with:
           ruby-version: ${{ matrix.ruby }}
-          bundler: 1
           apt-get: ragel
           brew: ragel
           mingw: _upgrade_ openssl ragel
@@ -49,13 +48,12 @@ jobs:
           if ('${{ matrix.ruby }}' -lt '2.3') {
             gem update --system 2.7.10 --no-document
           }
-          bundle install --jobs 4 --retry 3 --path=.bundle/puma
+          bundle install --jobs 4 --retry 3
 
       - name: compile
         run:  bundle exec rake compile
 
       - name: rubocop
-        if: startsWith(matrix.ruby, '2.')
         run: bundle exec rake rubocop
 
       - name: test

Gemfile

@@ -3,7 +3,7 @@ source "https://rubygems.org"
 gemspec
 
 gem "rdoc"
-gem "rake-compiler"
+gem "rake-compiler", "~> 0.9.4"
 
 gem "nio4r", "~> 2.0"
 gem "rack", "~> 1.6"

History.md

@@ -1,3 +1,9 @@
+### Master
+* Bugfixes
+  * Resolve issue with threadpool waiting counter decrement when thread is killed
+  * Constrain rake-compiler version to 0.9.4 to fix `ClassNotFound` exception when using MiniSSL with Java8.
+  * Ensure that TCP_CORK is usable
+
 ## 5.0.0
 
 * Features
@@ -6,11 +12,12 @@
   * EXPERIMENTAL: Added `nakayoshi_fork` option. Reduce memory usage in preloaded cluster-mode apps by GCing before fork and compacting, where available. (#2093, #2256)
   * Added pumactl `thread-backtraces` command to print thread backtraces (#2054)
   * Added incrementing `requests_count` to `Puma.stats`. (#2106)
-  * Increased maximum URI path length from 2048 to 8196 bytes (#2167)
+  * Increased maximum URI path length from 2048 to 8192 bytes (#2167, #2344)
   * `lowlevel_error_handler` is now called during a forced threadpool shutdown, and if a callable with 3 arguments is set, we now also pass the status code (#2203)
   * Faster phased restart and worker timeout (#2220)
   * Added `state_permission` to config DSL to set state file permissions (#2238)
   * Added `Puma.stats_hash`, which returns a stats in Hash instead of a JSON string (#2086, #2253)
+  * `rack.multithread` and `rack.multiprocess` now dynamically resolved by `max_thread` and `workers` respectively (#2288)
 
 * Deprecations, Removals and Breaking API Changes
   * `--control` has been removed. Use `--control-url` (#1487)
@@ -24,9 +31,11 @@
   * Daemonization has been removed without replacement. (#2170)
   * Changed #connected_port to #connected_ports (#2076)
   * Configuration: `environment` is read from `RAILS_ENV`, if `RACK_ENV` can't be found (#2022)
+  * Log binding on http:// for TCP bindings to make it clickable
 
 * Bugfixes
   * Fix JSON loading issues on phased-restarts (#2269)
+  * Improve shutdown reliability (#2312, #2338)
   * Close client http connections made to an ssl server with TLSv1.3 (#2116)
   * Do not set user_config to quiet by default to allow for file config (#2074)
   * Always close SSL connection in Puma::ControlCLI (#2211)
@@ -46,6 +55,8 @@
   * Fix `UserFileDefaultOptions#fetch` to properly use `default` (#2233)
   * Improvements to `out_of_band` hook (#2234)
   * Prefer the rackup file specified by the CLI (#2225)
+  * Fix for spawning subprocesses with fork_worker option (#2267)
+  * Set `CONTENT_LENGTH` for chunked requests (#2287)
 
 * Refactor
   * Remove unused loader argument from Plugin initializer (#2095)
@@ -58,6 +69,13 @@
   * Support parallel tests in verbose progress reporting (#2223)
   * Refactor error handling in server accept loop (#2239)
 
+## 4.3.4/4.3.5 and 3.12.5/3.12.6 / 2020-05-22
+
+Each patchlevel release contains a separate security fix. We recommend simply upgrading to 4.3.5/3.12.6.
+
+* Security
+  * Fix: Fixed two separate HTTP smuggling vulnerabilities that used the Transfer-Encoding header. CVE-2020-11076 and CVE-2020-11077.
+
 ## 4.3.3 and 3.12.4 / 2020-02-28
 
 * Bugfixes

README.md

@@ -8,7 +8,7 @@
 
 [![Code Climate](https://codeclimate.com/github/puma/puma.svg)](https://codeclimate.com/github/puma/puma)
 [![SemVer](https://api.dependabot.com/badges/compatibility_score?dependency-name=puma&package-manager=bundler&version-scheme=semver)](https://dependabot.com/compatibility-score.html?dependency-name=puma&package-manager=bundler&version-scheme=semver)
-[![StackOverflow](http://img.shields.io/badge/stackoverflow-Puma-blue.svg)]( http://stackoverflow.com/questions/tagged/puma )
+[![StackOverflow](https://img.shields.io/badge/stackoverflow-Puma-blue.svg)]( https://stackoverflow.com/questions/tagged/puma )
 
 Puma is a **simple, fast, multi-threaded, and highly concurrent HTTP 1.1 server for Ruby/Rack applications**.
 
@@ -27,7 +27,7 @@ $ gem install puma
 $ puma
 ```
 
-Without arguments, puma will look for a rackup (.ru) file in 
+Without arguments, puma will look for a rackup (.ru) file in
 working directory called `config.ru`.
 
 ## Frameworks
@@ -135,7 +135,7 @@ Preloading can’t be used with phased restart, since phased restart kills and r
 If puma encounters an error outside of the context of your application, it will respond with a 500 and a simple
 textual error message (see `lowlevel_error` in [this file](https://github.com/puma/puma/blob/master/lib/puma/server.rb)).
 You can specify custom behavior for this scenario. For example, you can report the error to your third-party
-error-tracking service (in this example, [rollbar](http://rollbar.com)):
+error-tracking service (in this example, [rollbar](https://rollbar.com)):
 
 ```ruby
 lowlevel_error_handler do |e|

docs/architecture.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-![http://bit.ly/2iJuFky](images/puma-general-arch.png)
+![https://bit.ly/2iJuFky](images/puma-general-arch.png)
 
 Puma is a threaded web server, processing requests across a TCP or UNIX socket.
 
@@ -12,7 +12,7 @@ Clustered mode is shown/discussed here. Single mode is analogous to having a sin
 
 ## Connection pipeline
 
-![http://bit.ly/2zwzhEK](images/puma-connection-flow.png)
+![https://bit.ly/2zwzhEK](images/puma-connection-flow.png)
 
 * Upon startup, Puma listens on a TCP or UNIX socket.
   * The backlog of this socket is configured (with a default of 1024), determining how many established but unaccepted connections can exist concurrently.
@@ -29,7 +29,7 @@ Clustered mode is shown/discussed here. Single mode is analogous to having a sin
 
 ### Disabling `queue_requests`
 
-![http://bit.ly/2zxCJ1Z](images/puma-connection-flow-no-reactor.png)
+![https://bit.ly/2zxCJ1Z](images/puma-connection-flow-no-reactor.png)
 
 The `queue_requests` option is `true` by default, enabling the separate thread used to buffer requests as described above.
 

docs/deployment.md

@@ -20,7 +20,10 @@ Welcome back!
 Puma was originally conceived as a thread-only webserver, but grew the ability to
 also use processes in version 2.
 
-Here are some rules of thumb:
+To run puma in single mode (e.g. for a development environment) you will need to
+set the number of workers to 0, anything above will run in cluster mode.
+
+Here are some rules of thumb for cluster mode:
 
 ### MRI
 
@@ -66,7 +69,8 @@ thread to become available.
 
 * Have your upstream proxy set a header with the time it received the request:
     * nginx: `proxy_set_header X-Request-Start "${msec}";`
-    * haproxy: `http-request set-header X-Request-Start "%t";`
+    * haproxy >= 1.9: `http-request set-header X-Request-Start t=%[date()]%[date_us()]`
+    * haproxy < 1.9: `http-request set-header X-Request-Start t=%[date()]`
 * In your Rack middleware, determine the amount of time elapsed since `X-Request-Start`.
 * To improve accuracy, you will want to subtract time spent waiting for slow clients:
     * `env['puma.request_body_wait']` contains the number of milliseconds Puma spent

docs/signals.md

@@ -1,8 +1,8 @@
-The [unix signal](http://en.wikipedia.org/wiki/Unix_signal) is a method of sending messages between [processes](http://en.wikipedia.org/wiki/Process_(computing)). When a signal is sent, the operating system interrupts the target process's normal flow of execution. There are standard signals that are used to stop a process but there are also custom signals that can be used for other purposes. This document is an attempt to list all supported signals that Puma will respond to. In general, signals need only be sent to the master process of a cluster.
+The [unix signal](https://en.wikipedia.org/wiki/Unix_signal) is a method of sending messages between [processes](https://en.wikipedia.org/wiki/Process_(computing)). When a signal is sent, the operating system interrupts the target process's normal flow of execution. There are standard signals that are used to stop a process but there are also custom signals that can be used for other purposes. This document is an attempt to list all supported signals that Puma will respond to. In general, signals need only be sent to the master process of a cluster.
 
 ## Sending Signals
 
-If you are new to signals it can be useful to see how they can be used. When a process is created in a *nix like operating system it will have a [PID - or process identifier](http://en.wikipedia.org/wiki/Process_identifier) that can be used to send signals to the process. For demonstration we will create an infinitely running process by tailing a file:
+If you are new to signals it can be useful to see how they can be used. When a process is created in a *nix like operating system it will have a [PID - or process identifier](https://en.wikipedia.org/wiki/Process_identifier) that can be used to send signals to the process. For demonstration we will create an infinitely running process by tailing a file:
 
 ```sh
 $ echo "foo" >> my.log
@@ -17,13 +17,13 @@ $ ps aux | grep tail
 schneems        87152   0.0  0.0  2432772    492 s032  S+   12:46PM   0:00.00 tail -f my.log
 ```
 
-You can send a signal in Ruby using the [Process module](http://www.ruby-doc.org/core-2.1.1/Process.html#kill-method):
+You can send a signal in Ruby using the [Process module](https://www.ruby-doc.org/core-2.1.1/Process.html#kill-method):
 
 ```
 $ irb
 > puts pid
 => 87152
-Process.detach(pid) # http://ruby-doc.org/core-2.1.1/Process.html#method-c-detach
+Process.detach(pid) # https://ruby-doc.org/core-2.1.1/Process.html#method-c-detach
 Process.kill("TERM", pid)
 ```
 

ext/puma_http11/http11_parser.c

@@ -14,12 +14,14 @@
 
 /*
  * capitalizes all lower-case ASCII characters,
- * converts dashes to underscores.
+ * converts dashes to underscores, and underscores to commas.
  */
 static void snake_upcase_char(char *c)
 {
     if (*c >= 'a' && *c <= 'z')
       *c &= ~0x20;
+    else if (*c == '_')
+      *c = ',';
     else if (*c == '-')
       *c = '_';
 }

ext/puma_http11/http11_parser.rl

@@ -12,12 +12,14 @@
 
 /*
  * capitalizes all lower-case ASCII characters,
- * converts dashes to underscores.
+ * converts dashes to underscores, and underscores to commas.
  */
 static void snake_upcase_char(char *c)
 {
     if (*c >= 'a' && *c <= 'z')
       *c &= ~0x20;
+    else if (*c == '_')
+      *c = ',';
     else if (*c == '-')
       *c = '_';
 }

ext/puma_http11/org/jruby/puma/MiniSSL.java

@@ -173,7 +173,7 @@ public IRubyObject initialize(ThreadContext threadContext, IRubyObject miniSSLCo
     engine.setEnabledProtocols(protocols);
     engine.setUseClientMode(false);
 
-    long verify_mode = miniSSLContext.callMethod(threadContext, "verify_mode").convertToInteger().getLongValue();
+    long verify_mode = miniSSLContext.callMethod(threadContext, "verify_mode").convertToInteger("to_i").getLongValue();
     if ((verify_mode & 0x1) != 0) { // 'peer'
         engine.setWantClientAuth(true);
     }

ext/puma_http11/puma_http11.c

@@ -54,7 +54,7 @@ DEF_MAX_LENGTH(FIELD_NAME, 256);
 DEF_MAX_LENGTH(FIELD_VALUE, 80 * 1024);
 DEF_MAX_LENGTH(REQUEST_URI, 1024 * 12);
 DEF_MAX_LENGTH(FRAGMENT, 1024); /* Don't know if this length is specified somewhere or not */
-DEF_MAX_LENGTH(REQUEST_PATH, 8196);
+DEF_MAX_LENGTH(REQUEST_PATH, 8192);
 DEF_MAX_LENGTH(QUERY_STRING, (1024 * 10));
 DEF_MAX_LENGTH(HEADER, (1024 * (80 + 32)));
 

lib/puma/binder.rb

@@ -6,14 +6,15 @@
 require 'puma/const'
 require 'puma/util'
 require 'puma/minissl/context_builder'
+require 'puma/configuration'
 
 module Puma
   class Binder
     include Puma::Const
 
     RACK_VERSION = [1,6].freeze
 
-    def initialize(events)
+    def initialize(events, conf = Configuration.new)
       @events = events
       @listeners = []
       @inherited_fds = {}
@@ -23,8 +24,8 @@ def initialize(events)
       @proto_env = {
         "rack.version".freeze => RACK_VERSION,
         "rack.errors".freeze => events.stderr,
-        "rack.multithread".freeze => true,
-        "rack.multiprocess".freeze => false,
+        "rack.multithread".freeze => conf.options[:max_threads] > 1,
+        "rack.multiprocess".freeze => conf.options[:workers] >= 1,
         "rack.run_once".freeze => false,
         "SCRIPT_NAME".freeze => ENV['SCRIPT_NAME'] || "",
 
@@ -113,7 +114,7 @@ def parse(binds, logger, log_msg = 'Listening')
                 i.local_address.ip_unpack.join(':')
               end
 
-              logger.log "* #{log_msg} on tcp://#{addr}"
+              logger.log "* #{log_msg} on http://#{addr}"
             end
           end
 

lib/puma/client.rb

@@ -308,8 +308,16 @@ def setup_body
 
       te = @env[TRANSFER_ENCODING2]
 
-      if te && CHUNKED.casecmp(te) == 0
-        return setup_chunked_body(body)
+      if te
+        if te.include?(",")
+          te.split(",").each do |part|
+            if CHUNKED.casecmp(part.strip) == 0
+              return setup_chunked_body(body)
+            end
+          end
+        elsif CHUNKED.casecmp(te) == 0
+          return setup_chunked_body(body)
+        end
       end
 
       @chunked_body = false
@@ -412,7 +420,10 @@ def read_chunked_body
           raise EOFError
         end
 
-        return true if decode_chunk(chunk)
+        if decode_chunk(chunk)
+          @env[CONTENT_LENGTH] = @chunked_content_length
+          return true
+        end
       end
     end
 
@@ -424,20 +435,28 @@ def setup_chunked_body(body)
       @body = Tempfile.new(Const::PUMA_TMP_BASE)
       @body.binmode
       @tempfile = @body
+      @chunked_content_length = 0
+
+      if decode_chunk(body)
+        @env[CONTENT_LENGTH] = @chunked_content_length
+        return true
+      end
+    end
 
-      return decode_chunk(body)
+    def write_chunk(str)
+      @chunked_content_length += @body.write(str)
     end
 
     def decode_chunk(chunk)
       if @partial_part_left > 0
         if @partial_part_left <= chunk.size
           if @partial_part_left > 2
-            @body << chunk[0..(@partial_part_left-3)] # skip the \r\n
+            write_chunk(chunk[0..(@partial_part_left-3)]) # skip the \r\n
           end
           chunk = chunk[@partial_part_left..-1]
           @partial_part_left = 0
         else
-          @body << chunk if @partial_part_left > 2 # don't include the last \r\n
+          write_chunk(chunk) if @partial_part_left > 2 # don't include the last \r\n
           @partial_part_left -= chunk.size
           return false
         end
@@ -484,12 +503,12 @@ def decode_chunk(chunk)
 
           case
           when got == len
-            @body << part[0..-3] # to skip the ending \r\n
+            write_chunk(part[0..-3]) # to skip the ending \r\n
           when got <= len - 2
-            @body << part
+            write_chunk(part)
             @partial_part_left = len - part.size
           when got == len - 1 # edge where we get just \r but not \n
-            @body << part[0..-2]
+            write_chunk(part[0..-2])
             @partial_part_left = len - part.size
           end
         else

lib/puma/cluster.rb

@@ -248,6 +248,7 @@ def worker(index, master)
       $0 = title
 
       Signal.trap "SIGINT", "IGNORE"
+      Signal.trap "SIGCHLD", "DEFAULT"
 
       fork_worker = @options[:fork_worker] && index == 0
 
@@ -284,9 +285,11 @@ def worker(index, master)
 
       if fork_worker
         restart_server.clear
+        worker_pids = []
         Signal.trap "SIGCHLD" do
-          Process.wait(-1, Process::WNOHANG) rescue nil
-          wakeup!
+          wakeup! if worker_pids.reject! do |p|
+            Process.wait(p, Process::WNOHANG) rescue true
+          end
         end
 
         Thread.new do
@@ -303,7 +306,7 @@ def worker(index, master)
             elsif idx == 0 # restart server
               restart_server << true << false
             else # fork worker
-              pid = spawn_worker(idx, master)
+              worker_pids << pid = spawn_worker(idx, master)
               @worker_write << "f#{pid}:#{idx}\n" rescue nil
             end
           end