You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Our prod uniquejobs:digests sorted set in redis grew to 3GB in about 3 weeks. (~5mil jobs/day, less than a 1000 total jobs in queues and dead job queue during screenshot time)
our lock TTLs are at max 6 hours - the vast majority are 5 minutes.
Expected behavior
my understanding is that we should clean the digests as conditions occur (mostly when our jobs exit successfully) or at worst when the reaper runs.
Current behavior
the uniquejobs:digests sorted set grows until we run out of redis ram
Worker class
47 different jobs that have a lock on them. the only locks we use are: until_and_while_executing, until_executing, & until_executed, 95% of them are until_and_while_executing
# our entire sidekiq configrequire'sidekiq'require'sidekiq-unique-jobs'Sidekiq.default_job_options={'backtrace'=>true,'retry'=>15}Sidekiq.strict_args!SidekiqUniqueJobs.configuredo |config|
config.lock_info=trueconfig.lock_prefix='prod_uniq'# new valueconfig.lock_ttl=5.minutes# default for anything - any longer jobs should have one specifiedconfig.enabled= !Rails.env.test?config.logger_enabled=falseconfig.debug_lua=falseconfig.max_history=10_000config.reaper=:ruby# :ruby, :lua or :none/nilconfig.reaper_count=50# Stop reaping after this many keysconfig.reaper_interval=305# Reap orphans every 5 minutesconfig.reaper_timeout=30endSidekiq.default_configuration.redis={url: ENV['REDIS_URL'] || 'redis://localhost:6379/0',network_timeout: 3}# relax redis timeouts a bit default is 1Sidekiq.configure_serverdo |config|
ifconfig.queues == ['default']concurrency=(ENV['SIDEKIQ_CONCURRENCY'] || (Rails.env.development? ? 7 : 23)).to_iconfig.queues=%w[af,10o,8ws,5r,5s,4t,4sl,3searchkick,2sd,1c,1]config.concurrency=concurrencyconfig.capsule('limited')do |cap|
cap.concurrency=Rails.env.production? ? (concurrency / 3) : 1cap.queues=%w[af,10wms,4t,4searchkick,3sd,1]endconfig.capsule('single')do |cap|
cap.concurrency=1cap.queues=%w[counters,1]endendconfig.client_middlewaredo |chain|
chain.addSidekiqUniqueJobs::Middleware::Clientendconfig.server_middlewaredo |chain|
chain.addSidekiqUniqueJobs::Middleware::Serverendconfig.logger.level=ENV.fetch('SIDEKIQ_LOG_LEVEL',Logger::INFO)ifRails.env.production?SidekiqUniqueJobs::Server.configure(config)endSidekiq.configure_clientdo |config|
config.client_middlewaredo |chain|
chain.addSidekiqUniqueJobs::Middleware::Clientendend
Additional context
We're generally running the top of main from a version perspective. currently 8.0.6. sidekiq 7.1.6 currently, rails 7.0.8.
This is the second or 3rd time that we've seen the issue cropped up, not sure if it's been introduced recently or if it's always been there and we just haven't noticed until recently.
Failures or jobs exiting because of an exception or other 'non normal' exit are less than 0.1% of all jobs run.
I've been through the reaper issues, found some similar issues but seemingly nothing exact.
As always, huge love for the gem <3
The text was updated successfully, but these errors were encountered:
Describe the bug
Our prod uniquejobs:digests sorted set in redis grew to 3GB in about 3 weeks. (~5mil jobs/day, less than a 1000 total jobs in queues and dead job queue during screenshot time)
our lock TTLs are at max 6 hours - the vast majority are 5 minutes.
Expected behavior
my understanding is that we should clean the digests as conditions occur (mostly when our jobs exit successfully) or at worst when the reaper runs.
Current behavior
the uniquejobs:digests sorted set grows until we run out of redis ram
Worker class
47 different jobs that have a lock on them. the only locks we use are:
until_and_while_executing
,until_executing
, &until_executed
, 95% of them areuntil_and_while_executing
Additional context
We're generally running the top of main from a version perspective. currently 8.0.6. sidekiq 7.1.6 currently, rails 7.0.8.
This is the second or 3rd time that we've seen the issue cropped up, not sure if it's been introduced recently or if it's always been there and we just haven't noticed until recently.
Failures or jobs exiting because of an exception or other 'non normal' exit are less than 0.1% of all jobs run.
I've been through the reaper issues, found some similar issues but seemingly nothing exact.
As always, huge love for the gem <3
The text was updated successfully, but these errors were encountered: