Memory
Sidekiq uses memory, it's true. You'll see memory consumed booting Rails, loading your application code and running your jobs. The smallest Sidekiq process runs around 30 MB, a typical Sidekiq process running Rails and your application will use 300-500 MB but it's not unusual to see processes using 1-2 GB.
NOTE: The Ruby VM controls memory allocation and usage. Sidekiq is pure Ruby and simply boots Rails and your application code. Memory leaks and bloat can come from ANY code in your Gemfile and application. 90% of the time the cause is your application code, 10% of the time the issue is in Rails or another gem in your Gemfile. Please don't open an issue with Sidekiq for memory problems unless you can show proof that Sidekiq is the cause.
RSS (Resident Set Size) is the amount of physical RAM consumed by a process. Typically a Sidekiq process will start at some relatively small value and reach some steady state value where RAM usage remains flat. If your RSS never stops growing, that can indicate you have a memory leak somewhere!
On Linux, glibc clashes very badly with Ruby's multithreading and bloats quickly. Set MALLOC_ARENA_MAX=2.
If your process balloons up quickly upon processing some job, that can mean a job is loading a lot of data into memory. It's common for a poorly written ActiveRecord query to load many thousands of rows into memory. Use find_in_batches when querying for large resultsets.
Ruby's GC has been a performance sore spot in the past but isn't as much of a problem with recent releases. Don't touch any GC config unless you have good reason.
- Set
MALLOC_ARENA_MAX=2. This environment variable is the closest thing to a silver bullet if you are using Linux/glibc in production. On Heroku,heroku config:set MALLOC_ARENA_MAX=2 - Use derailed_benchmarks to find memory hotspots.
- Switch to
jemalloc. Ruby 2.7 works great with jemalloc 5.x.
Here's a collection of really useful blog posts about the Ruby VM, GC and memory. Nate Berkopec is the expert in profiling, memory usage and optimization if your company needs help optimizing a memory hungry app.