Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster mode: Client will not send READONLY command when reconnecting to replicas #1017

Closed
johnduhart opened this issue Jul 6, 2021 · 4 comments

Comments

@johnduhart
Copy link

After switching our application to use a cluster-mode redis setup, we discovered that (nearly) all reads to replicas were being redirected to the primary instance. Our application is an API, running in puma with multiple workers (ie. multiple forked processes).

After reviewing the library code, our current theory is that after forking the connection is forced to be re-established because of the process ID change (code). However, because the READONLY command is only sent when the client is initially created, the new connection is not in the correct state to allow for read only commands.

A potential fix for this would be to introduce a new Connector implementation for cluster mode clients that sends the READONLY command after connection.

@byroot
Copy link
Collaborator

byroot commented Jul 6, 2021

the READONLY command is only sent when the client is initially created

That's a bug indeed. That READONLY should be moved inside the client, and re-issued on reconnect.

@byroot byroot closed this as completed in ae80708 Jul 7, 2021
@byroot
Copy link
Collaborator

byroot commented Jul 7, 2021

@johnduhart I pushed ae80708, could you test the master branch just to make sure I haven't overlooked anything.

If that fix your issue as expected I can cut a release.

@johnduhart
Copy link
Author

I'll try to get this deployed into a testing environment today and report back.

@johnduhart
Copy link
Author

@byroot I deployed with ae80708 and everything looks good, no more MOVED errors reported by our telemetry and the number of reads on the primary instance is near-zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants