fork_worker automatic refork never triggers

[slondr]

Describe the bug

I have fork_worker enabled and was excited to deprecate my hand-spun automatic cluster restart with the built-in automatic refork functionality. However, after a couple months of using fork_worker live on production, I have never seen a refork actually occur without manual intervention. On a typical day, my application gets about 40,000 hits, and I have puma configured for a cluster size of 8 workers. I've tried setting the value of the fork_worker to as low as 150, which should mean a median of (40k requests divided by 8 workers divided by 1/150 for a time series of reforks per time quanta...) 21 reforks per minute! Yet, I never see the telltale drop in memory usage that I see when I manually send SIGURG to the cluster, never see any of the typical worker startup messages in the logs, etc.

Puma config:

environment 'production'
workers 8
threads 1, 7
fork_worker 150
plugin :tmp_restart
bind 'unix:///path/to/server.sock'
activate_control_app 'unix://path/to/socket.sock', auth_token: 'hello world'
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }

I launch puma via systemd, with the command RAILS_ENV=production bundle exec --keep-file-descriptors puma.

[MSP-Greg]

@slondr

I think more data might be helpful. I set fork_worker to 1000, and threw a wrk stream at a server, and it did restart, so I'm not sure what the issue is (yet). That was using master, not a release.

21 reforks per minute!

You might check that, I think the number should be more like a refork once every 43 minutes? 40k/(8*24) = 208 req per hour per worker...

[slondr]

Oh yeah, worth mentioning this is on puma 5.6.4 — though I don't see anything in the 5.6.5 release notes that I think would change this? I'll go test that + master with my application to see if it does.

[MSP-Greg]

@slondr

Most of my local test code is for master; I probably should have checked 5.6.5 or 5.6.4. I'll check a release sometime soon...

[slondr]

I was looking into this more today and checked my pumactl stats. It looks like workers 1 through 7 each have thousands of requests served but worker 0 only has 21. I'm guessing that's why I never see a refork. It's weird that this is so consistent though. I don't really know how Puma's scheduling system works for cluster mode but this definitely doesn't seem right to me.

[MSP-Greg]

Interesting. I'll see if I can repo.

[dentarg]

@slondr still having this issue? are you able to reproduce it for us if so?

[rus-max]

This will enable automatic fork

# Auto-fork after the specified number of requests.
if (fork_requests = @options[:fork_worker].to_i) > 0
  @events.register(:ping!) do |w|
    fork_worker! if w.index == 0 &&
      w.last_status[:requests_count] >= fork_requests * (w.phase + 1)
  end
end

[rus-max]

Auto-fork after the specified number of requests always. Commit