Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xds: add Outlier Detection Balancer #5413

Closed
wants to merge 5 commits into from
Closed

Conversation

zasweq
Copy link
Contributor

@zasweq zasweq commented Jun 7, 2022

This PR adds the Outlier Detection Balancer as part of the xDS configured tree of balancers (see https://github.com/grpc/proposal/blob/master/A50-xds-outlier-detection.md).

Contains #5371.

RELEASE NOTES:

  • xDS: add support for Outlier Detection

@zasweq zasweq added this to the 1.48 Release milestone Jun 7, 2022
@zasweq zasweq requested review from easwars and dfawley June 7, 2022 22:21
@dfawley dfawley changed the title xds: Added Outlier Detection Balancer xds: add Outlier Detection Balancer Jun 8, 2022
@@ -106,3 +107,43 @@ func (s) TestClientSideXDS(t *testing.T) {
t.Fatalf("rpc EmptyCall() failed: %v", err)
}
}

func (s) TestOutlierDetection(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we please move this test to be in a file on its own. Otherwise over time, this file is going to become a dumping ground for tests. Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

Comment on lines +125 to +131
resources := e2e.DefaultClientResources(e2e.ResourceParams{
DialTarget: serviceName,
NodeID: nodeID,
Host: "localhost",
Port: port,
SecLevel: e2e.SecurityLevelNone,
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these resource even contain appropriate outlier detection configuration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does. xdsclient will emit a nil Outlier Detection struct, which will cause the cds balancer to prepare a "no-op" Outlier Detection config. I verified it was present in the system manually through logs. Let me know if you want an explicit check.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the test fail if the CDS balancer suddenly decides to not prepare a "no-op" Outlier Detection config? If so, how?

if _, err := client.EmptyCall(ctx, &testpb.Empty{}, grpc.WaitForReady(true)); err != nil {
t.Fatalf("rpc EmptyCall() failed: %v", err)
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, what really are we testing here with regards to outlier detection?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doug mentioned to me he wanted a simple e2e sanity test with Outlier Detection as part of the system. More specific e2e functionality will come as part of the interop tests, which Michael is currently writing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But what sanity is this test testing? What change can cause this test to fail? And when it does fail, how does it help with identifying the issue that caused the failure?

@easwars easwars assigned zasweq and unassigned easwars and dfawley Jun 16, 2022
@easwars
Copy link
Contributor

easwars commented Jun 16, 2022

Not sure if all changes suggested on the other PR have made it here. For example, I don't see the balancer registration being protected with an environment variable. So, I'll probably hold off reviewing this until you let me know that this is ready.

Copy link
Contributor Author

@zasweq zasweq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will rebase after getting to your other comments on other PR.

Comment on lines +125 to +131
resources := e2e.DefaultClientResources(e2e.ResourceParams{
DialTarget: serviceName,
NodeID: nodeID,
Host: "localhost",
Port: port,
SecLevel: e2e.SecurityLevelNone,
})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does. xdsclient will emit a nil Outlier Detection struct, which will cause the cds balancer to prepare a "no-op" Outlier Detection config. I verified it was present in the system manually through logs. Let me know if you want an explicit check.

if _, err := client.EmptyCall(ctx, &testpb.Empty{}, grpc.WaitForReady(true)); err != nil {
t.Fatalf("rpc EmptyCall() failed: %v", err)
}
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doug mentioned to me he wanted a simple e2e sanity test with Outlier Detection as part of the system. More specific e2e functionality will come as part of the interop tests, which Michael is currently writing.

@@ -106,3 +107,43 @@ func (s) TestClientSideXDS(t *testing.T) {
t.Fatalf("rpc EmptyCall() failed: %v", err)
}
}

func (s) TestOutlierDetection(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

@zasweq
Copy link
Contributor Author

zasweq commented Jun 17, 2022

Rebased and got to your comments in a new PR, #5435.

@dfawley dfawley closed this Jun 28, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants