Client hitting ConnectionPool::TimeoutError in Sidekiq 7 #5702
Comments
|
Please supply the full backtrace of the error. |
|
Sure. Here's the backtrace: |
|
Sidekiq logs when it actually instantiates a connection pool: logger&.info { "Sidekiq #{Sidekiq::VERSION} connecting to Redis with options #{scrub(symbolized_options)}" }Add a |
|
The connection pool is timing out after 1 sec. It shouldn't take more than a few ms to push a job to Redis unless you have something very unusual going on. You need to get backtraces from all threads in the process to determine why 5 threads are actively holding a Redis connection. |
|
Added a log and can confirm the connection pool is being created after the block runs. I should have stressed how sporadic this has been happening as well. There were 4 occurrences in the last 3 days. |
|
And if you change to size: 10, the log still says 5? |
Yes. I never changed it back it. It still has |
|
Please post your entire initializer. |
Sidekiq::Client.reliable_push! unless Rails.env.test?
Sidekiq::Enterprise.unique! unless Rails.env.test?
Sidekiq.strict_args!
Sidekiq.configure_client do |config|
config.redis = {
host: ENV.fetch("REDIS_HOST"),
password: ENV.fetch("REDIS_PASSWORD"),
size: 10
}
end
Sidekiq.configure_server do |config|
config.redis = {host: ENV.fetch("REDIS_HOST"), password: ENV.fetch("REDIS_PASSWORD")}
config.logger.formatter = Sidekiq::Logger::Formatters::Base.new
config.super_fetch!
config.reliable_scheduler!
end |
|
Move the first three lines to the bottom. |
|
For the sake of completeness I also added back the log line. Here's the full initializer after the update: Sidekiq.configure_client do |config|
Rails.logger.warn("Sidekiq client config initialized")
config.redis = {
host: ENV.fetch("REDIS_HOST"),
password: ENV.fetch("REDIS_PASSWORD"),
size: 10
}
end
Sidekiq.configure_server do |config|
config.redis = {host: ENV.fetch("REDIS_HOST"), password: ENV.fetch("REDIS_PASSWORD")}
config.logger.formatter = Sidekiq::Logger::Formatters::Base.new
config.super_fetch!
config.reliable_scheduler!
end
Sidekiq::Client.reliable_push! unless Rails.env.test?
Sidekiq::Enterprise.unique! unless Rails.env.test?
Sidekiq.strict_args!And the resulting log: |
Ruby version: 3.1.3
Rails version: 7.0.4
Sidekiq / Pro / Enterprise version(s): Sidekiq 7.0.2 / Pro 7.0.5
We are seeing very sporadic
ConnectionPool:TimeoutErrorhappening in our client web/puma process.Since the default concurrency changed from 10 to 5 in Sidekiq 7 (and we didn't face this issue before the upgrade) our first instinct was to up the size of the client connection pool to 10.
Relevant parts of our initializer:
However adding
size: 10in the initializer had no effect. From what I can gather the client is using the "internal" connection pool which seems to be hard-coded to always have the size of 5 regardless of configuration. And since in Sidekiq 7 these pools are managed internally creating a different connection pool is not an option.I should add that we are very far from hitting either CPU or connections limit in our Redis instance.
Is the client connection pool size not meant to be configurable in this way?
The text was updated successfully, but these errors were encountered: