Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconnect on known errors after failover when pushing jobs to Redis #5159

Merged
merged 1 commit into from Feb 2, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/sidekiq.rb
Expand Up @@ -103,6 +103,7 @@ def self.redis
# to disconnect and reopen the socket to get back to the primary.
# 4495 Use the same logic if we have a "Not enough replicas" error from the primary
# 4985 Use the same logic when a blocking command is force-unblocked
# The same retry logic is also used in client.rb
if retryable && ex.message =~ /READONLY|NOREPLICAS|UNBLOCKED/
conn.disconnect!
retryable = false
Expand Down
19 changes: 17 additions & 2 deletions lib/sidekiq/client.rb
Expand Up @@ -189,8 +189,23 @@ def enqueue_in(interval, klass, *args)

def raw_push(payloads)
@redis_pool.with do |conn|
conn.pipelined do |pipeline|
atomic_push(pipeline, payloads)
retryable = true
begin
conn.pipelined do |pipeline|
atomic_push(pipeline, payloads)
end
rescue Redis::BaseError => ex
# 2550 Failover can cause the server to become a replica, need
# to disconnect and reopen the socket to get back to the primary.
# 4495 Use the same logic if we have a "Not enough replicas" error from the primary
# 4985 Use the same logic when a blocking command is force-unblocked
# The retry logic is copied from sidekiq.rb
if retryable && ex.message =~ /READONLY|NOREPLICAS|UNBLOCKED/
conn.disconnect!
retryable = false
retry
end
raise
end
end
true
Expand Down