Ent Multi Process

Sidekiq Enterprise has the ability to start and manage multiple Sidekiq child processes, like Unicorn or Puma. With multi-process mode, you get several advantages:

1. modest memory savings by sharing memory between processes
2. running multiple processes on a single Heroku dyno, allowing you to minimize your dyno costs
3. easy to create a single service in Upstart or Systemd which scales to all machine cores
4. automated memory monitoring and restart for bloated child processes

The term for running multi-process is swarm. A swarm has a parent process and N child processes.

Starting a Swarm

Sidekiq Enterprise provides a sidekiqswarm binary. This binary is designed to run under Upstart, Systemd or Foreman as a service. It does not allow old-style options like --daemonize, --logfile or --pidfile.

sidekiqswarm [options]

Start and supervise a swarm of Sidekiq processes.
All arguments are passed to each Sidekiq instance.

You may not use the `-d`, `-L` or `-P` options.

Use the SIDEKIQ_* environment variables to control sidekiqswarm.

SIDEKIQ_COUNT	        Number of Sidekiq child processes to start, defaults to number of cores
SIDEKIQ_MAXMEM_MB	Max RSS size in MB of child process before the parent will restart it
SIDEKIQ_PRELOAD         Comma-separated list of Bundler groups to preload before forking
SIDEKIQ_PRELOAD_APP	Preload application for increased memory savings

Example:
SIDEKIQ_COUNT=5 SIDEKIQ_MAXMEM_MB=300 bundle exec sidekiqswarm -r ./myworker.rb

Heroku

Change sidekiq to sidekiqswarm in your Procfile:

worker: SIDEKIQ_MAXMEM_MB=1700 bundle exec sidekiqswarm

Add a SIDEKIQ_MAXMEM_MB variable like above if you want to gracefully restart bloated Sidekiq processes. If you are using -c, do not set it higher than 10.

Note the swarm feature is not useful on 1x and 2x dynos which don't have enough memory to run multiple large app processes. It has a HUGE positive impact on Performance-L dynos because P-L dynos have eight CPUs: a Sidekiq process will only utilize 12.5% of a P-L dyno whereas swarm will utilize 100%. Since P-L dynos have 8 cores and 14GB of memory, note SIDEKIQ_MAXMEM_MB: 8 * 1700MB ~= 14GB

Running via init

systemd

Sidekiq has a sample systemd unit file here. Starting sidekiqswarm instead is almost identical, just update the ExecStart line and configure the environment as necessary:

# if you want to override the default number of processes
Environment=SIDEKIQ_COUNT=2
ExecStart=/usr/local/bin/bundle exec sidekiqswarm -e production

upstart

Sidekiq has a sample upstart conf file here. Starting sidekiqswarm instead is almost identical, just update the exec line within the script block and configure the environment as necessary:

# if you want to override the default number of processes
env SIDEKIQ_COUNT=2

exec bundle exec sidekiqswarm -e production

Signals and Controlling a Swarm

Use the standard upstart and systemd tools to manage the service for your swarm, e.g. systemctl restart sidekiq.

You can send the USR2, TERM and TSTP signals to the parent process and it will pass those signals to the underlying children. Once the parent process has received USR2, TSTP or TERM, it will not spawn any more children; it must be restarted. The parent process does not handle the TTIN signal.

Bundler Preload

Sidekiq forks the child processes after running Bundler.require(:default) so the children can share the memory consumed by loading the gems. Your Gemfile should eager load gems where possible; using gem 'something', require: false in your Gemfile will limit any memory savings.

If you find that sidekiqswarm's default Bundler require is breaking your app on boot, you can control which groups get preloaded or disable preload completely:

# preload both the default and production groups
SIDEKIQ_PRELOAD=default,production bundle exec sidekiqswarm ...
# disable gem preload completely
SIDEKIQ_PRELOAD= bundle exec sidekiqswarm ...

The most frequent reason why preload breaks is if a gem requires Rails to already be loaded when it is required. You can fix this by preloading Rails explicitly:

gem "rails", require: "rails/all"

This supercedes the require at the top of config/application.rb. See #4766 for a discussion on preload breakage.

Application Preload

As of Sidekiq Enterprise 2.1.1, sidekiqswarm can preload your entire application before forking each child process, leading to substantial memory savings (20-30% is not uncommon). App preload can be dangerous so it is opt-in.

SIDEKIQ_PRELOAD_APP=1 bundle exec sidekiqswarm ...

If your application initializers use network connections, spawn threads or create shared Mutexes, this can lead to problems. You can define "after fork" callbacks to clean up these resources, e.g.:

Sidekiq.configure_server do |config|
  config.on(:fork) do
    ActiveRecord::Base.clear_active_connections!
  end
end

If you see weird behavior with preload enabled and normal behavior without it, search the issues and open a new issue if you don't find anything relevant.

Memory Monitoring

The parent process can watch all children and restart any that get above a certain memory usage. Set the SIDEKIQ_MAXMEM_MB environment variable with the maximum memory in megabytes. If a child goes over that limit, the parent will detect it and do the following:

1. Send USR2 to child
2. Child will stop pulling new jobs, finish existing jobs and exit (this is the rolling restart feature)
3. Parent waits for the child to exit and forks a new child

Please see the Problems and Troubleshooting page for more tips on taming process RSS.