Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis Cluster not recognizing new cluster nodes with identical hostnames #1778

Open
annervisser opened this issue Jun 27, 2023 · 0 comments · May be fixed by #1779
Open

Redis Cluster not recognizing new cluster nodes with identical hostnames #1778

annervisser opened this issue Jun 27, 2023 · 0 comments · May be fixed by #1779

Comments

@annervisser
Copy link

annervisser commented Jun 27, 2023

We're in the process of testing MemoryDB and it's failure cases. We've run into an issue when upgrading a cluster while our application is using ioredis.

Environment

  • Server: AWS MemoryDB
  • Cluster: 1 master 2 slaves
  • ioredis: 5.3.1

Problem

When changing the instance type of a MemoryDB cluster, an in-place upgrade happens where new nodes are deployed with the same hostnames as the old ones. The old ones start returning MOVED to all queries, and after a while they are removed.

The way ioredis reacts to this is it ends up in a loop of retrying commands and receiving MOVED. This loop is only broken once the old nodes are removed and the individual connections time out.

Cause

The MOVED responses cause ioredis to refresh it's slots, which ends up resetting the connectionpool with the new slots:

this.connectionPool.reset(nodes);

But because the hostname and port are identical to the old nodes1, no new connections are made and the old ones are left intact. 2

Solution

I believe this could be fixed by including the nodeId (the third return value from CLUSTER SLOTS3) in the node keys, so that a new id will trigger a new connection.

Footnotes

  1. https://github.com/redis/ioredis/blob/d1d9dba9eafc574a9d9041fd4bc7cd04f1584159/lib/cluster/util.ts#L25-L29

  2. https://github.com/redis/ioredis/blob/06782e681500eae6f3ceafcc6385b9be4fdaf4e3/lib/cluster/ConnectionPool.ts#L125-L135

  3. https://redis.io/commands/cluster-slots/

annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to annervisser/ioredis that referenced this issue Jun 27, 2023
annervisser added a commit to ScientaNL/ioredis that referenced this issue Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant