New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sidekiqswarm: Add support for preloading Rails app #4646
Comments
I'm unclear what you are suggesting.
1 shouldn't be too hard; you can already do it today with |
Yes, that's what I'm trying to suggest. For my setting at least, the gems preloaded by I'm not sure how to evaluate the risks of this approach -- each process should have their own copy of all objects (copy-on-write), what might go wrong if Rails was booted before forking? |
The issue is that any locks, threads or sockets created in Rails initializers will be shared by all child processes. That's the purpose of |
That'd be great! Thank you for explaining, I see how this can be tricky now. For apps that do not require re-initializing, this can be a big win. Maybe there can be an |
No guarantees but I'll investigate. |
I'm implementing this right now. Here's the before and after memory data I see on OSX. Before
After
Notice the multiple children drop from 80MB to 40MB, the single parent swarm process grows from 40MB to 80MB. This is on a trivial default Rails app; that's massive savings. As for an Sidekiq.configure_server do |config|
config.on(:fork) do
# ActiveRecord::Base.clear_active_connections! or whatever
end
end |
Awesome! This is great! |
https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process#application-preload 2.1.1 will be released soon. |
@justin808 Impossible for me to know. There are always CPU time and memory space tradeoffs in software. That's literally what a cache is. |
Ruby version: 2.5.5
Sidekiq / Pro / Enterprise version(s): Sidekiq 5.1.3 / Pro 4.0.3 / Ent 1.7.2
Context
In my project, with a fresh irb process
consumes 272MB memory.
Repro
Start a sidekiq swarm with 30 processes. The processes consume 10GB (> 30 * 272 MB) memory.
Investigation
I looked into
/proc/<pid>/smaps
and aggregated the memory segments:About ~9GB (30 * 272 MB) of the segments are under Private Dirty.
I patched it so it calls
boot_system
before forking the child processes. With the patch,The private mem segs only take ~0.62GB, and the overall memory usage dropped to ~1.5GB.
Thoughts on this approach?
The text was updated successfully, but these errors were encountered: