-
Notifications
You must be signed in to change notification settings - Fork 11
Heroku
Using sidekiq with Heroku is simple, use Procfile
with your Rails app to start a worker process:
web: bundle exec puma ...
worker: bundle exec sidekiq -c 10
See this page for more details about Procfiles, Foreman and Heroku.
WARNING: don't ever use a concurrency value greater than 10 without thorough testing. I've seen dozens of customers crushing their dyno CPUs with concurrency: 25
or higher. This will only make jobs slower.
To connect to your Redis addon, you'll need to tell Sidekiq which environment variable name to use by setting REDIS_PROVIDER to the name of that variable, e.g. heroku config:set REDIS_PROVIDER=REDISTOGO_URL
. Make sure you're using discrete Redis instances for any other purposes such as a Rails cache store. Lacking the REDIS_PROVIDER variable Sidekiq can also use the REDIS_URL environment variable.
Heroku Redis 6.0 and later on production tiers requires TLS encryption but uses self-signed certificates. With the default SSL configuration, connections will fail with a self-signed-certificate error. This behavior can be configured for Sidekiq in an initializer file. See Using Redis: Using an Initializer.
An example:
SIDEKIQ_REDIS_CONFIGURATION = {
url: ENV[ENV["REDIS_PROVIDER"]], # if one assumes that REDIS_PROVIDER indirection is reliably present
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }, # we must trust Heroku and AWS here
}
Sidekiq.configure_server do |config|
config.redis = SIDEKIQ_REDIS_CONFIGURATION
end
Sidekiq.configure_client do |config|
config.redis = SIDEKIQ_REDIS_CONFIGURATION
end
Earlier versions of Heroku Redis use stunnel to secure their connection, in conjunction with the heroku/redis
buildpack; this requires no custom configuration in Sidekiq, but the buildpack will attempt to stunnel all Heroku Redis connections, including any Redis 6 instances. It is much simpler to keep all Redis instances on the later generation.
The redis
gem has supported the verify_mode
key since version 4.0.2. The hiredis
gem does not work in this environment, so do not include it in your Gemfile if you are using redis
with rediss://
and OpenSSL::SSL::VERIFY_NONE
.
Setting this environment variable will greatly reduce Sidekiq's memory usage and is highly recommended:
heroku config:set MALLOC_ARENA_MAX=2
As of September 2019 Heroku changed MALLOC_ARENA_MAX
to default to 2
for new Ruby applications. Ruby apps created after then shouldn't need to set this environment variable.
I don't recommend setting Sidekiq's concurrency greater than 10. At larger values, you will see poor performance due to CPU contention. With 1x
and 2x
dynos, I would set concurrency to 5.
Sidekiq will pay attention to a few well-known environment variables:
- REDIS_PROVIDER - should contain the name of the environment variable for your chosen Redis provider
- RAILS_ENV - "production", "staging", "development", alternative to
-e
- RAILS_MAX_THREADS - controls process concurrency, alternative to
-c
The best value dyno types for Rails apps are the 2x
and Performance-L
dynos. I don't suggest using 1x
or Performance-M
dynos unless you have a specific reason. If you are using a Performance-L
dyno, you won't utilize anywhere near 100% of the dyno unless you are using Sidekiq Enterprise's Multi-Process feature, which will dramatically improve Sidekiq's concurrency and allow it to crunch through 8x as many jobs (since each P-L dyno has 8 cores).
worker: SIDEKIQ_MAXMEM_MB=1750 bundle exec sidekiqswarm ...
With SIDEKIQ_MAXMEM_MB, the swarm parent process will restart any Sidekiq child processes which bloat past 1.75GB of memory. Since P-Ls have 14GB of memory (8 * 1.75 = 14), this should keep the entire thing within RAM with no more R14 memory errors.
Don't tune the pool manually. All modern connection pools in Ruby are lazy, they will use as many connections as they need but no more. Lower Sidekiq's concurrency if you need to use fewer connections.
Heroku puts a hard limit of 30 seconds on a process restart, Sidekiq's default of -t 25
gives jobs 25 seconds to finish before starting the "force shutdown" procedure. After 25 seconds, Sidekiq will push any remaining unfinished jobs back to Redis and exit.
Make sure your jobs are idempotent so they will safely restart when the process starts back up.
You can optionally use require 'sidekiq/api';Sidekiq::ProcessSet.new.each(&:quiet!)
in a pre-deploy hook to give your jobs even more time to finish. If your pre-deploy process fails for some reason, you must still terminate Sidekiq using Sidekiq::ProcessSet.new.each(&:stop!)
. Heroku will start new processes and they will start fetching jobs again. See #5047.
If you need to get backtraces from a running Heroku worker process, you can use the Sidekiq API to deliver a TTIN signal:
❯ bundle exec rails c
irb(main):001:0> require 'sidekiq/api'
irb(main):003:0> Sidekiq::ProcessSet.new.each {|ps| ps.dump_threads }
Within 5 seconds, the process should log a backtrace for every thread in the process.
Home | The Basics | Best Practices | Using Redis | Error Handling | Advanced Options | Problems?
This wiki is tracked by git and publicly editable. You are welcome to fix errors and typos. Any defacing or vandalism of content will result in your changes being reverted and you being blocked.