Add option to avoid workers crash-looping

[grosser]

We run fluentd (which uses serverengine) in a container, sometimes the workers keep dying in a tight loop, which puts lots of stress on the system, but it's not visible to the outside since the server process just keeps restarting the workers.

I'd like to make a PR to add a "max_crash_frequency" or so flag, that would crash the server if the worker crash frequency goes above a certain value (like 10/minute or so)

/cc @repeatedly @tagomoris

Monkeypatch atm is:

# frozen_string_literal: true
# changes serverengine/lib/serverengine/multi_process_server.rb to crash the server when workers fail too often
#
# https://github.com/treasure-data/serverengine/issues/96

module PreventWorkerCrashloop
  MAX_WORKER_CRASHES = 5
  MAX_WORKER_CRASH_INTERVAL = 5 * 60

  def alive?
    alive = super

    if !alive && !@unrecoverable_exit
      now = Time.now.to_i
      cutoff = now - MAX_WORKER_CRASH_INTERVAL
      failures = (@@failures_timestamps ||= []) # rubocop:disable Style/ClassVars
      failures.reject! { |t| t < cutoff }
      failures << now
      if failures.size >= MAX_WORKER_CRASHES
        diff = now - failures.first
        @worker.logger.error(
          "PreventWorkerCrashloop killing server because of #{failures.size} worker crashes in #{diff}s"
        )
        @unrecoverable_exit = true
      end
    end

    alive
  end
end

ServerEngine::MultiProcessServer::WorkerMonitor.prepend(PreventWorkerCrashloop)