Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xCluster] Deadlock during Setup from BG thread AutoFlag validation #22376

Closed
1 task done
hari90 opened this issue May 13, 2024 · 0 comments
Closed
1 task done

[xCluster] Deadlock during Setup from BG thread AutoFlag validation #22376

hari90 opened this issue May 13, 2024 · 0 comments
Assignees
@hari90
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/highest Highest priority issue

Comments

@hari90
Copy link
Contributor

hari90 commented May 13, 2024 •
edited by jira bot

Jira Link: DB-11277

Description

HandleLocalAutoFlagsConfigChange and RefreshAutoFlagConfigVersion take the ClusterConfig write lock before ReplicationInfo write lock. Whereas AddCDCStreamToUniverseAndInitConsumer take the locks in the other order.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@hari90 hari90 added the area/docdb YugabyteDB core features label May 13, 2024
@hari90 hari90 self-assigned this May 13, 2024
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue priority/highest Highest priority issue and removed priority/medium Medium priority issue labels May 13, 2024
hari90 added a commit that referenced this issue May 15, 2024
@hari90
[#22376] xCluster: Acquire locks in correct order when handling AutoF…
d9deed4 
…lagsConfigChange

Summary:
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D35072
hari90 added a commit that referenced this issue May 15, 2024
@hari90
[BACKPORT 2.20][#22376] xCluster: Acquire locks in correct order when…
7a64e88 
… handling AutoFlagsConfigChange

Summary:
Original commit: d9deed4 / D35072
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D35077
hari90 added a commit that referenced this issue May 15, 2024
@hari90
[BACKPORT 2024.1][#22376] xCluster: Acquire locks in correct order wh…
a8e6586 
…en handling AutoFlagsConfigChange

Summary:
Original commit: d9deed4 / D35072
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D35076
@hari90 hari90 closed this as completed May 16, 2024
svarnau pushed a commit that referenced this issue May 25, 2024
@hari90
[#22376] xCluster: Acquire locks in correct order when handling AutoF…
8461063 
…lagsConfigChange

Summary:
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D35072
svarnau pushed a commit that referenced this issue May 29, 2024
@hari90
[BACKPORT 2024.1][#22376] xCluster: Acquire locks in correct order wh…
9a30c7f 
…en handling AutoFlagsConfigChange

Summary:
Original commit: d9deed4 / D35072
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D35076
svarnau pushed a commit that referenced this issue May 30, 2024
@hari90
[BACKPORT 2.20][#22376] xCluster: Acquire locks in correct order when…
06942be 
… handling AutoFlagsConfigChange

Summary:
Original commit: d9deed4 / D35072
`HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion` take the ClusterConfig write lock before ReplicationInfo write lock. Whereas `AddCDCStreamToUniverseAndInitConsumer` take the locks in the other order causing a deadlock of setup replication.

Fixed the lock order in `HandleLocalAutoFlagsConfigChange` and `RefreshAutoFlagConfigVersion`.
Jira: DB-11277

Test Plan: Jenkins

Reviewers: slingam, jhe, xCluster, asrivastava

Reviewed By: asrivastava

Subscribers: ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D35077
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hari90 hari90

Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/highest Highest priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hari90 @yugabyte-ci