phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

gingerlime · 2020-10-29T12:53:42Z

Describe the bug

This seems to happen when doing phased-restarts using versioned deploy folders with symlinks.

We use a folder structure similar to this:

/app/app.d/{version (int)}
/app/app -> /app/app.d/{version} # symlink
/app/prev -> /app/app.d/{version-1} # symlink

When we deploy, we increment the version, run bundle etc, then switch the symlink and do a phased-restart of puma.

All seems to work fine, but after a few deploys, we also start pruning older versions, and then we see this error in the puma.log

/usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in `block in materialize': Could not find json-2.3.1 in any of the sources (Bundler::GemNotFound)
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in `map!'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in `materialize'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in `specs'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/definition.rb:237:in `specs_for'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/definition.rb:226:in `requested_specs'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:101:in `block in definition_method'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:20:in `setup'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler.rb:149:in `setup'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in `block in <top (required)>'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:136:in `with_level'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:88:in `silence'
        from /usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in `<top (required)>'
        from /usr/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:92:in `require'
        from /usr/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:92:in `require'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/configuration.rb:312:in `rack_builder'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/configuration.rb:331:in `load_rackup'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/configuration.rb:257:in `app'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/runner.rb:142:in `app'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/runner.rb:146:in `start_server'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster/worker.rb:57:in `run'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:195:in `worker'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:96:in `block in spawn_worker'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:96:in `fork'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:96:in `spawn_worker'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:78:in `block in spawn_workers'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:71:in `times'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:71:in `spawn_workers'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:142:in `check_workers'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cluster.rb:410:in `run'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/launcher.rb:171:in `run'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/lib/puma/cli.rb:80:in `run'
        from /var/local/app/app.d/1386/vendor/bundle/ruby/2.6.0/gems/puma-5.0.4/bin/puma-wild:25:in `<main>'
[12769] + Gemfile in context: /var/local/app/app.d/1391/Gemfile
[12769] Early termination of worker

❗️ Note that the app now lives in /app/app.d/1390 but the errors are pointing to puma in /app/app.d/1386 ❗️

Puma config:

#!/usr/bin/env puma

# The directory to operate out of.
#
# The default is the current directory.
#
directory "/var/local/app/app"

# Set the environment in which the rack's app will run. The value must be a string.
#
# The default is "development".
#
environment ENV["RAILS_ENV"] || "development"

# Store the pid of the server in the file at "path".
#
pidfile "/var/run/puma/puma.pid"

# Use "path" as the file to store the server info state. This is
# used by "pumactl" to query and control the server.
#
state_path "/var/run/puma/puma.state"

# Redirect STDOUT and STDERR to files specified. The 3rd parameter
# ("append") specifies whether the output is appended, the default is
# "false".
#
stdout_redirect "/var/log/puma.log", "/var/log/puma.log", true

# Configure "min" to be the minimum number of threads to use to answer
# requests and "max" the maximum.
#
# The default is "0, 16".
#
# threads 0, 16
threads 8, 8


# Bind the server to "url". "tcp://", "unix://" and "ssl://" are the only
# accepted protocols.
#
# The default is "tcp://0.0.0.0:9292".
#
bind 'unix:///var/run/puma/puma.sock'


# === Cluster mode ===

# How many worker processes to run.  Typically this is set to
# to the number of available cores.
#
# The default is "0".
#
workers ENV["PUMA_WORKERS"] || 16

# Allow workers to reload bundler context when master process is issued
# a USR1 signal. This allows proper reloading of gems while the master
# is preserved across a phased-restart. (incompatible with preload_app)
# (off by default)

# see https://github.com/puma/puma/blob/master/docs/deployment.md#restarting
prune_bundler

If I do a full restart, then things are back to normal, until it happens again (we keep 3 deploys back, so after 4 deploys...)

To Reproduce

I was only able to reproduce this on our staging environment so far

Expected behavior
phased-restart should reload everything from the new folder... It does seem to point to the right folder, but there's some mysterious left-overs pointing to something old somewhere??

Server:

OS: Ubuntu 18.04 with systemd
Puma Version 5.0.4 (it doesn't happen with 3.12)

The text was updated successfully, but these errors were encountered:

nateberkopec · 2020-10-29T13:11:12Z

/usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in `block in materialize': Could not find json-2.3.1 in any of the sources (Bundler::GemNotFound)

For some reason I remember bringing this up during the nio4r change but I was told that json wasn't an issue?

dentarg · 2020-10-29T13:13:27Z

From #2018 (comment)

I opened #2427 which fixes the nio4r issue with regard to phased restarts. It does not allow one to remove from disk compiled native extensions for other gems that are loaded by the puma master process (like ffi)

Is Puma master process still using json because of the following?

puma/lib/puma/cluster/worker_handle.rb

Lines 43 to 47 in f0dffd4

    
           def ping!(status) 
        
             @last_checkin = Time.now 
        
             require 'json' 
        
             @last_status = JSON.parse(status, symbolize_names: true) 
        
           end

puma/lib/puma/cluster.rb

Lines 441 to 442 in f0dffd4

    
           w.ping!(result.sub(/^\d+/,'').chomp) 
        
           @launcher.events.fire(:ping!, w)

rmacklin · 2020-10-29T14:05:09Z

/usr/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in `block in materialize': Could not find json-2.3.1 in any of the sources (Bundler::GemNotFound)

For some reason I remember bringing this up during the nio4r change but I was told that json wasn't an issue?

For reference, here's that discussion: #2427 (comment)

MSP-Greg · 2020-10-29T14:22:46Z

This issue is about running bundle/setup, and hence, is not being triggered by any require of a gem by Puma.

I'm not sure yet, but Puma needs json, but it's not in a gemspec/Gemfile. Also, not sure about the environment, but Ruby 2.6.x's default json is 2.1.0, and the log refers to the current json version of 2.3.1

@gingerlime does your app use json?

gingerlime · 2020-10-29T14:27:49Z

@MSP-Greg yes, I'm pretty sure it does (I would imagine most apps these days would?)

I can find it in our Gemfile.lock file, although I'm not sure where the requirement comes from (it's not in our Gemfile)

EDIT: I can see a gem that requires json (algoliasearch), plus a few other gems that require multi_json in our app

MSP-Greg · 2020-10-29T14:35:12Z

@gingerlime

Thanks,

No need to figure what's dependent on it. I need to drink more coffee and think about this. I assume json-2.3.1 is installed somewhere in your app bundle?

gingerlime · 2020-10-29T14:40:16Z

I assume json-2.3.1 is installed somewhere in your app bundle?

Yes, exactly. This is the version we have installed in our Gemfile.lock file

Enjoy your ☕ and thanks for your help!

MSP-Greg · 2020-10-29T14:48:26Z

Puma has the following in its gemspec

s.add_runtime_dependency "nio4r", "~> 2.0"

We do not have a similar line for json. I'm not sure if that's the issue.

cjlarose · 2020-10-29T14:52:31Z

This is essentially the same issue as #2018 and the behavior is not especially surprising to me.

From https://github.com/puma/puma/blob/d4d1ed374a72d0df020d9512707b9eea17344135/docs/restart.md

If you remove the gems from old releases as part of your deployment strategy, there are additional considerations. Do not put any gems into extra_runtime_dependencies that have native extensions or have dependencies that have native extensions (one common example is puma_worker_killer and its dependency on ffi). Workers will fail on boot during a phased restart. The underlying issue is recorded in an issue on the rubygems project. Hot restarts are your only option here if you need these dependencies.

If the puma master process has activated any gem that has native extensions (such as nio4r, ffi, or json), and the deployment process removes the gems from disk, phased restarts will fail.

@dentarg is essentially right here. I think the change to include json in the master process in #1801 is the culprit and I think deferring when to load it as we did in #2269 is insufficient. We cannot have the json gem present in the master process ever (such as in Puma::Cluster::WorkerHandler#ping!).

cjlarose · 2020-10-29T14:54:56Z

I should mention, too, that if a solution is found for rubygems/rubygems#4004, then we can safely use the json gem in the puma master process. But although that change alone would make it so that users can safely remove gems on disk, they still wouldn't be able to upgrade the json gem (I think this is a separate issue present in puma still). Once the puma master process loads a gem, the application must use the same version of that gem.

With or without a fix for rubygems/rubygems#4004, I think our best move forward is to remove the use of the json gem from the puma master process.

MSP-Greg · 2020-10-29T15:13:28Z

@cjlarose

our best move forward is to remove the use of the json gem from the puma master process.

Regardless, should the following be added to the gemspec? If it's no longer optional...

s.add_runtime_dependency "json", "~> 2.3"

cjlarose · 2020-10-29T15:15:17Z

@MSP-Greg I think the json gem is bundled with every version of Ruby that puma supports, so an explicit runtime dependency declaration for it is not necessary. I'm not 100% sure, but there's some discussion about it in #1801

MSP-Greg · 2020-10-29T15:21:27Z

@cjlarose

I'm not 100% sure

My problem also.

It's not a matter of whether it exists, it's a matter of whether/how it gets resolved (which is the issue here), and runtime dependencies are handled in a specific matter. Especially when they are shared by the app, which is less likely with nio4r... I think.

cjlarose · 2020-10-29T15:25:11Z

I'll start by seeing if I can write a red test for this bug.

I don't know what the fix will be yet since we probably don't want to re-introduce the malformed JSON bug that was fixed by #1801 .

Removing the use of JSON from various stats commands is likely a breaking change (that's public API)
Adding a new runtime dependency on a pure-Ruby JSON serialization/deserialization library would make it so that users can remove old gems from disk, but would create a new problem where users could not upgrade that new pure Ruby gem.
Rolling our own JSON serialization/deserialization has proved to be error-prone

cjlarose · 2020-10-29T15:31:48Z

@MSP-Greg

It's not a matter of whether it exists, it's a matter of whether/how it gets resolved (which is the issue here), and runtime dependencies are handled in a specific matter. Especially when they are shared by the app, which is less likely with nio4r... I think.

If you're thinking about how runtime_dependencies are handled by prune_bundler, we actually removed all that stuff in #2427. Essentially, since the puma master process no longer has any dependencies on any gems (except those bundled by Ruby itself), preserving the runtime_dependencies in the new process after the exec operation performed by prune_bundler is no longer necessary.

MSP-Greg · 2020-10-29T16:20:33Z

@cjlarose

No, I'm thinking about how the last Puma line leading to the error in the above log is (v5.0.4):

puma/lib/puma/configuration.rb

Line 312 in 1f0c08e

require 'bundler/setup'

It's not throwing the exception on a require 'json' statement.

cjlarose · 2020-10-29T17:01:08Z

@cjlarose

No, I'm thinking about how the last Puma line leading to the error in the above log is (v5.0.4):

puma/lib/puma/configuration.rb

Line 312 in 1f0c08e

require 'bundler/setup'

It's not throwing the exception on a require 'json' statement.

I see. I think you're on to something with the add_runtime_dependency call. I added an integration test to ensure that it is possible to upgrade/downgrade the json gem in an application as part of a phased restart (this fails). What I found is that without the add_runtime_dependency in Puma's gemspec (basically what happens on master now), we get a bunch of constant redefinition warnings when workers boot.

/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/version.rb:4: warning: already initialized constant JSON::VERSION
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/version.rb:4: warning: previous definition of VERSION was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/version.rb:5: warning: already initialized constant JSON::VERSION_ARRAY
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/version.rb:5: warning: previous definition of VERSION_ARRAY was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/version.rb:6: warning: already initialized constant JSON::VERSION_MAJOR
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/version.rb:6: warning: previous definition of VERSION_MAJOR was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/version.rb:7: warning: already initialized constant JSON::VERSION_MINOR
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/version.rb:7: warning: previous definition of VERSION_MINOR was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/version.rb:8: warning: already initialized constant JSON::VERSION_BUILD
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/version.rb:8: warning: previous definition of VERSION_BUILD was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/common.rb:100: warning: already initialized constant JSON::NaN
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:107: warning: previous definition of NaN was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/common.rb:102: warning: already initialized constant JSON::Infinity
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:109: warning: previous definition of Infinity was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/common.rb:104: warning: already initialized constant JSON::MinusInfinity
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:111: warning: previous definition of MinusInfinity was here
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/2.7.0/json/common.rb:129: warning: already initialized constant JSON::UnparserError
/Users/chris.larose/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:136: warning: previous definition of UnparserError was here

I think this indicates that the JSON gem is being double-loaded (it's loaded once by the puma master process and loaded again by the workers). Usually require is supposed to prevent double-loading, so there's something strange going on. I suspect that when the workers require 'json', they're referencing a different path to load the gem, so Ruby happily loads it up into memory, clobbering the JSON module that is already loaded into memory.

I'll spend more time digging into this, but I suspect @MSP-Greg 's hunch that we should add the add_runtime_dependency declaration is correct. That'll force the puma master process to use the version of the json gem that is declared in the users' application's Gemfile.lock, if available, instead of using the one bundled by Ruby. Doing that alone doesn't fix OP's original problem, but it's a good first step.

cjlarose · 2020-10-30T04:26:36Z

I was able to create a minimal repro for @gingerlime's issue in Docker

https://github.com/cjlarose/puma-phased-restart-could-not-find-gem-errors/tree/could-not-find-json

It demonstrates that if you have a deployment process that deletes gems from old releases and your application has a dependency on the json gem (declared in the application Gemfile or otherwise activated by RubyGem's gem method), workers will fail on boot repeatedly with Could not find json-2.3.1 in any of the sources.

I'm still investigating the precise mechanism that causes it.

h0jeZvgoxFepBQ2C · 2020-10-30T09:26:12Z

I can confirm this error too ( we also have fork_worker enabled).

MSP-Greg · 2020-10-30T16:05:58Z

@cjlarose

Messy. I took some of the code you mentioned above. Running four tests, upgrade/downgrade by json/nio4r.

I had to shut off warnings when requiring, as reloading code will always show the constant warnings.

But, the downgrade json test is always failing (in a big way). Also, json and nio4r have to explicitly be added to the 'project' Gemfile. So, I'm still working on it. The branch in my fork is https://github.com/MSP-Greg/puma/tree/json. There's two commits, the first renames test files, etc. The second (most recent) is the actual code changes...

cjlarose · 2020-10-30T22:54:59Z

Ok so I took a look at all of the places where we explicitly use the json gem in puma. There are four places where we require 'json', all listed neatly in #2269. Let's look at them individually:

The worker `stat payload` thread

puma/lib/puma/cluster/worker.rb

Lines 114 to 115 in d4d1ed3

    
           require 'json' 
        
           io << "p#{Process.pid}#{server.stats.to_json}\n"

This usage of require 'json' is safe. It necessarily happens inside the worker process. If the user's application depends on json, it'll use that version. Otherwise, it'll use the version of json bundled with Ruby.

`WorkerHandle#ping!`

puma/lib/puma/cluster/worker_handle.rb

Lines 43 to 48 in d4d1ed3

    
           def ping!(status) 
        
             @last_checkin = Time.now 
        
             require 'json' 
        
             @last_status = JSON.parse(status, symbolize_names: true) 
        
           end

This use of require 'json' is not safe. Assuming prune_bundler is on, the first time this is called will be after the first worker has booted. That worker will check in to the master process and the master process will try to parse the message. Under the normal RubyGems require logic, if the user's application defined a dependency on the json gem, the puma master process will add that version to the $LOAD_PATH, activate the json gem's gemspec, and load the puma gem into memory. If the user's application did not define such a dependency, this will load the version of the json gem that's bundled with Ruby.

Subsequent attempts to boot workers will try to use the same version as the version that was first require'd. As a consequence, so long as the application declares a dependency on the json gem, removing that first version's representation on disk will cause workers to fail, even if they're booting up a version of the application that depends on the same version of json (they didn't upgrade or downgrade). The root cause of this particular problem is rubygems/rubygems#4004. The other problem is that once the json gem is loaded into the puma master process, it is no longer possible to upgrade or downgrade the json gem at all in a subsequent phased restart (this is true even if you don't delete old versions of your application). In short, our best path forward is to remove the json gem entirely from being loaded into the master process.

The use of the JSON gem in this particular method was introduced in #2124 and can be safely reverted. Doing so will probably fix @gingerlime's problem and allow many other folks to upgrade to a modern version of Puma. The remaining two places where we require 'json' probably affect fewer people, but the impact is just as severe.

Status server requests to `/gc-stats`, `/stats`, and `/thread-backtraces`.

puma/lib/puma/app/status.rb

Lines 25 to 27 in d4d1ed3

    
           if env['PATH_INFO'] =~ /\/(gc-stats|stats|thread-backtraces)$/ 
        
             require 'json' 
        
           end

puma/lib/puma/app/status.rb

Lines 51 to 62 in d4d1ed3

    
           when 'gc-stats' 
        
             GC.stat.to_json 
        
           when 'stats' 
        
             @launcher.stats.to_json 
        
           when 'thread-backtraces' 
        
             backtraces = [] 
        
             @launcher.thread_status do |name, backtrace| 
        
               backtraces << { name: name, backtrace: backtrace } 
        
             end 
        
             backtraces.to_json

This usage of require 'json' is similarly unsafe. As soon as json is required into the puma master process, it's possible for subsequent worker boot operations to fail (either because the native extensions for an old release were deleted or because the version of json has changed in the new version of the application).

The use of the json gem in the status server was introduced in #1801 in order to fix a bug with malformed responses from /gc-stats on JRuby. At the time of authorship of #1801, /stats did not rely on the json gem and /thread-backtraces did not yet exist.

/thread-backtraces is a fairly recent addition from #2054. At the time of authorship, it just depended on the json gem having already been loaded.

It is probably possible to fix the original problem of malformed JSON with /gc-stats without loading the json gem. It would just mean that we'd have to implement and maintain basically a hand-rolled JSON serialization implementation.

For /stats, it's possible to implement this without the json gem. In fact, before #2086, it didn't use the json gem at all--just hand-rolled JSON serialization.

For /thread-backtraces, this can be implemented with some more hand-rolled JSON serialization, but it never has been in Puma's history (like I said, this one is fairly recent).

Puma.stats

puma/lib/puma.rb

Lines 27 to 30 in d4d1ed3

    
           def self.stats 
        
             require 'json' 
        
             @get_stats.stats.to_json 
        
           end

This use of require 'json' is similarly unsafe for the same reasons as before.

Puma.stats returns a JSON-encoded string. It is considered public API.

#2086 modified this method to just return a Hash instead of a String, but that was later rolled back in #2253 because it was a breaking API change. Interestingly, the revert operation in #2253 effectively deferred JSON serialization until the last moment (instead of Cluster#stats returning a JSON-encoded string as it did before #2086, it returns a hash). Conversion to a JSON string happens in the body of Puma.stats.

Similar to the /stats in the discussion about the status server, there was a time when this method did not depend on the json gem, and we can roll it back with some more hand-rolled JSON serialization.

cjlarose · 2020-10-30T23:04:46Z

I'll start with removing the use of the json gem in WorkerHandle#ping! since I think that'll have an immediate impact for most folks that experience this problem. The other fixes I suggested can be follow-up PRs.

cjlarose · 2020-11-03T00:49:31Z

#2473 (and some follow-up work) has been merged. I'll take a look at implementing the fixes for the status server and for Puma.stats.

cjlarose · 2020-11-03T07:15:04Z

Opened #2479 to fix the remaining uses of the json gem.

cjlarose · 2020-11-10T17:28:46Z

@gingerlime If you're able to test your application against puma's master branch, we'd appreciate finding out if it works for you! Otherwise, we can wait 'till the next puma release. We appreciate your patience and can't wait for you to be able to upgrade to puma 5.x!

gingerlime · 2020-11-11T06:22:31Z

@cjlarose yes, of course. it's the least I could do :) I'll get it tested on our staging environment so we can go through a number of deploys and hopefully spot any issues. I'll report back within a day or two at the latest.

gingerlime · 2020-11-11T15:35:24Z

@cjlarose we've been running puma master all day on our staging environment, with a dozen or so deploys all using phased restarts. So far it looks great 👍 🚀

I'm not sure we really tested all scenarios, like Gem upgrades/new installs/removals though, but I'm definitely optimistic about the upgrade to 5.x now.

Thanks so much for all your hard work and patience.

gingerlime · 2020-11-17T10:29:38Z

~~I know this isn't related, but thought I'd ask before opening a separate issue (considering we're still not "officially" using 5.x and I'm just testing master on our staging environment).~~

We noticed a weird issue with our sidekiq processes suddenly crashing and failing to restart. Looking closer, it seems related to a problem with bootsnap and some file ownership... The bootsnap cache file/folder ownership in our /var/local/app/app/tmp/cache folder seems to switch to root somehow when we phase-restart puma ... on our live environment (with puma 3.x) this doesn't seem to happen though. Is it possibly related to the puma version upgrade? (both our live and staging environment are using latest bootsnap 1.5.1)

EDIT: Please ignore. Seems completely unrelated to puma 5.x vs 3.x

gingerlime mentioned this issue Oct 29, 2020

phased-restart: Could not find nio4r-2.5.1 in any of the sources (Bundler::GemNotFound) #2018

Closed

nateberkopec added bug restart labels Oct 29, 2020

nateberkopec assigned cjlarose Oct 30, 2020

cjlarose mentioned this issue Oct 31, 2020

Remove use of json gem from WorkerHandle#ping! #2473

Merged

8 tasks

cjlarose mentioned this issue Nov 3, 2020

Remove use of json gem #2479

Merged

8 tasks

nateberkopec closed this as completed in #2479 Nov 10, 2020

gingerlime mentioned this issue Apr 10, 2022

Bundler error prevents phased-restarting puma after upgrading to rails 7 #2843

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

gingerlime commented Oct 29, 2020

nateberkopec commented Oct 29, 2020

dentarg commented Oct 29, 2020

rmacklin commented Oct 29, 2020

MSP-Greg commented Oct 29, 2020

gingerlime commented Oct 29, 2020 •

edited

MSP-Greg commented Oct 29, 2020

gingerlime commented Oct 29, 2020 •

edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 •

edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 30, 2020

h0jeZvgoxFepBQ2C commented Oct 30, 2020

MSP-Greg commented Oct 30, 2020 •

edited

cjlarose commented Oct 30, 2020

cjlarose commented Oct 30, 2020 •

edited

cjlarose commented Nov 3, 2020

cjlarose commented Nov 3, 2020

cjlarose commented Nov 10, 2020 •

edited

gingerlime commented Nov 11, 2020

gingerlime commented Nov 11, 2020

gingerlime commented Nov 17, 2020 •

edited

phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

Comments

gingerlime commented Oct 29, 2020

nateberkopec commented Oct 29, 2020

dentarg commented Oct 29, 2020

rmacklin commented Oct 29, 2020

MSP-Greg commented Oct 29, 2020

gingerlime commented Oct 29, 2020 • edited

MSP-Greg commented Oct 29, 2020

gingerlime commented Oct 29, 2020 • edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 • edited

cjlarose commented Oct 29, 2020 • edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 • edited

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 • edited

cjlarose commented Oct 29, 2020

MSP-Greg commented Oct 29, 2020

cjlarose commented Oct 29, 2020 • edited

cjlarose commented Oct 30, 2020

h0jeZvgoxFepBQ2C commented Oct 30, 2020

MSP-Greg commented Oct 30, 2020 • edited

cjlarose commented Oct 30, 2020

The worker stat payload thread

WorkerHandle#ping!

Status server requests to /gc-stats, /stats, and /thread-backtraces.

Puma.stats

cjlarose commented Oct 30, 2020 • edited

cjlarose commented Nov 3, 2020

cjlarose commented Nov 3, 2020

cjlarose commented Nov 10, 2020 • edited

gingerlime commented Nov 11, 2020

gingerlime commented Nov 11, 2020

gingerlime commented Nov 17, 2020 • edited

gingerlime commented Oct 29, 2020 •

edited

gingerlime commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

cjlarose commented Oct 29, 2020 •

edited

MSP-Greg commented Oct 30, 2020 •

edited

The worker `stat payload` thread

`WorkerHandle#ping!`

Status server requests to `/gc-stats`, `/stats`, and `/thread-backtraces`.

cjlarose commented Oct 30, 2020 •

edited

cjlarose commented Nov 10, 2020 •

edited

gingerlime commented Nov 17, 2020 •

edited