Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow Adding Client Level Attributes to MetricsTracerFactory #2614

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

lqiu96
Copy link
Contributor

@lqiu96 lqiu96 commented Mar 29, 2024

Allow the MetricsTracerFactory to take in a second parameter (Map of attributes) that will be treated as client level attributes. These attributes will be added to every single MetricsTracer created throughout the lifecycle of the client.

Was able to verify this behavior inside Cloud Monitoring:
image

Additional Attribute was recorded.

Via:

    InstantiatingGrpcChannelProvider channelProvider =
            InstantiatingGrpcChannelProvider.newBuilder().build();

    Map<String, String> clientAttributesMapping = new HashMap<>();
    clientAttributesMapping.put("directpath_enabled", String.valueOf(channelProvider.canUseDirectPath()));

    ...
    options
      .setApiTracerFactory(new MetricsTracerFactory(recorder, clientAttributesMapping))
      .build();

@product-auto-label product-auto-label bot added the size: s Pull request size is small. label Mar 29, 2024
@lqiu96 lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 29, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 29, 2024
@lqiu96 lqiu96 requested a review from blakeli0 April 1, 2024 19:07
@lqiu96 lqiu96 marked this pull request as ready for review April 1, 2024 19:07
@lqiu96 lqiu96 requested a review from a team as a code owner April 1, 2024 19:07
@lqiu96 lqiu96 marked this pull request as draft April 2, 2024 20:47
@product-auto-label product-auto-label bot added size: l Pull request size is large. size: m Pull request size is medium. and removed size: s Pull request size is small. size: l Pull request size is large. labels Apr 2, 2024
@lqiu96 lqiu96 force-pushed the directpath-compatible-getter branch from 0d9781f to 05b1cde Compare April 2, 2024 21:10
Comment on lines 69 to 73
MetricsTracer metricsTracer =
new MetricsTracer(
MethodName.of(spanName.getClientName(), spanName.getMethodName()), metricsRecorder);
for (Map.Entry<String, String> attributeEntrySet : attributes.entrySet()) {
metricsTracer.addAttributes(attributeEntrySet.getKey(), attributeEntrySet.getValue());
}
return metricsTracer;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible options:

  1. Use MetricTracer's addAttributes() to add only add client level attributes. The clientName and methodName attributes will continue to be passed via MethodName via the constructor.
  2. Add MethodName attribute with addAttributes() and refactor MetricsTracer to only take in metricsRecorder

Probably going to keep it this way (option 1).

Comment on lines 62 to 65
public MetricsTracerFactory(MetricsRecorder metricsRecorder, Map<String, String> attributes) {
this.metricsRecorder = metricsRecorder;
this.attributes = attributes;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional constructor to take in user configured in client level attributes.

In the future, we can have Gax automatically collect attributes and this would need a constructor to take in a set of attribute flags. Class is marked with InternalApi and could be open to more refactoring if we want to pursue this.

@lqiu96 lqiu96 changed the title feat: Add DirectPath Compatible Getter to gRPC channel provider feat: Allow Adding Client Level Attributes to MetricsTracerFactory Apr 9, 2024
@lqiu96 lqiu96 marked this pull request as ready for review April 9, 2024 21:11
@lqiu96 lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label Apr 12, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Apr 12, 2024
@lqiu96 lqiu96 requested a review from blakeli0 April 12, 2024 21:04
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Apr 16, 2024
"Echo.Echo",
MetricsTracer.LANGUAGE_ATTRIBUTE,
MetricsTracer.DEFAULT_LANGUAGE,
"directpath_enabled",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we extract the String literals that are used twice to variables?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

@@ -117,6 +126,7 @@ public final class InstantiatingGrpcChannelProvider implements TransportChannelP
@Nullable private final Boolean allowNonDefaultServiceAccount;
@VisibleForTesting final ImmutableMap<String, ?> directPathServiceConfig;
@Nullable private final MtlsProvider mtlsProvider;
private final SystemProductNameReader systemProductNameReader;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally we thought exposing a default scope method would be good enough, but now that we are introducing a new field and a new private class just for testing, I'm not sure it is worth it. I'm now actually leaning towards keep isOnComputeEngine() as it is.
In the future, we can introduce mockito-inline or mockito v5(requires Java 11) that can mock/spy final classes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bit of change added just for testing. I think there is quite a bit of added benefit, namely that we get to test multiple functions and also to get to test how they interact together:

@Test
public void canUseDirectPath_happyPath() {
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isTrue();
}
@Test
public void canUseDirectPath_directPathEnvVarDisabled() {
EnvironmentProvider envProvider = Mockito.mock(EnvironmentProvider.class);
Mockito.when(
envProvider.getenv(
InstantiatingGrpcChannelProvider.DIRECT_PATH_ENV_DISABLE_DIRECT_PATH))
.thenReturn("true");
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_directPathEnvVarNotSet_attemptDirectPathIsTrue() {
EnvironmentProvider envProvider = Mockito.mock(EnvironmentProvider.class);
// If system property is not set, then System.getProperty() returns null
Mockito.when(
envProvider.getenv(
InstantiatingGrpcChannelProvider.DIRECT_PATH_ENV_DISABLE_DIRECT_PATH))
.thenReturn(null);
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isTrue();
}
@Test
public void canUseDirectPath_directPathEnvVarNotSet_attemptDirectPathIsFalse() {
EnvironmentProvider envProvider = Mockito.mock(EnvironmentProvider.class);
Mockito.when(
envProvider.getenv(
InstantiatingGrpcChannelProvider.DIRECT_PATH_ENV_DISABLE_DIRECT_PATH))
.thenReturn(null);
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(false)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_nonComputeCredentials() {
Credentials credentials = Mockito.mock(Credentials.class);
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(credentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_systemProductNameInvalid() throws IOException {
InstantiatingGrpcChannelProvider.SystemProductNameReader systemProductNameReader =
Mockito.mock(InstantiatingGrpcChannelProvider.SystemProductNameReader.class);
Mockito.when(systemProductNameReader.getSystemProductName()).thenReturn("testing");
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_systemPropertyIsNotLinux() throws IOException {
System.setProperty("os.name", "Windows");
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_systemProductNameThrowsIOException() throws IOException {
InstantiatingGrpcChannelProvider.SystemProductNameReader systemProductNameReader =
Mockito.mock(InstantiatingGrpcChannelProvider.SystemProductNameReader.class);
Mockito.when(systemProductNameReader.getSystemProductName()).thenThrow(new IOException());
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(DEFAULT_ENDPOINT)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
@Test
public void canUseDirectPath_nonGDUUniverseDomain() {
String nonGDUEndpoint = "test.random.com:443";
InstantiatingGrpcChannelProvider provider =
InstantiatingGrpcChannelProvider.newBuilder()
.setEnvProvider(envProvider)
.setAttemptDirectPath(true)
.setCredentials(computeEngineCredentials)
.setSystemProductNameReader(systemProductNameReader)
.setEndpoint(nonGDUEndpoint)
.build();
Truth.assertThat(provider.canUseDirectPath()).isFalse();
}
. I believe the behavior is kept the same and I am adding wrappers for the same functionality.

The added classes and methods are all package-private scope and should not impact the customers at all. And since they are package-private, we should be able to easily remove and refactor to use mockito v5 for this same purpose.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic is the same but the behavior is slightly different. isOnComputeEngine() used to be a static method, now it is not because we have to initialize an instance of SystemProductNameReader. It would be great if we can keep isOnComputeEngine() static. Or even better, isOnComputeEngine() does not have to be a static method, it could be a static block so that it does not get executed every time we call it. The benefit is marginal though since InstantiatingGrpcChannelProvider should only be called during client initialization.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I have overlooked the intention behind isOnComputeEngine() being static. Let me see if I can find a way around this (either with keeping it static or moving it to a static clock).

I would also prefer to not change the behavior if possible.

Copy link

sonarcloud bot commented Apr 23, 2024

Quality Gate Failed Quality Gate failed for 'gapic-generator-java-root'

Failed conditions
69.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

Copy link

sonarcloud bot commented Apr 23, 2024

Quality Gate Failed Quality Gate failed for 'java_showcase_integration_tests'

Failed conditions
55.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

* @return if DirectPath is enabled for the client AND if the configurations are valid
*/
@InternalApi
public boolean canUseDirectPath() {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add a check for isDirectPathXdsEnabled

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohanli-ml I believe you had helped implement this logic before. We're trying to expose a getter for the conditions that would enable DirectPath for this gRPC channel. Should isDirectPathXdsEnabled() be added here?

I copied over the original configs set:

if (isDirectPathEnabled()
&& isCredentialDirectPathCompatible()
&& isOnComputeEngine()
&& canUseDirectPathWithUniverseDomain()) {

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are conditions that we have canUseDirectPath is true but isDirectPathXdsEnabled is false based on the current logic, maybe we can expose isDirectPathXdsEnabled as a public method, and the Spanner team can set a client level attribute based on canUseDirectPath() && isDirectPathXdsEnabled()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make isDirectPathXdsEnabled() public with @InternalApi annotation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@surbhigarg92 Would you be fine with the changes above?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lqiu96 LGTM.

@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. size: xl Pull request size is extra large. labels May 22, 2024
@product-auto-label product-auto-label bot added the size: l Pull request size is large. label May 22, 2024
@lqiu96 lqiu96 force-pushed the directpath-compatible-getter branch from 3c21f02 to 8b17e83 Compare May 22, 2024 19:22
@lqiu96
Copy link
Contributor Author

lqiu96 commented May 22, 2024

Error:

Error:  Failures: 
Error:    OpencensusTracerFactoryTest.testImplicitParentSpan:102 
Wanted but not invoked:
internalTracer.spanBuilderWithExplicitParent(
    <any string>,
    same(Mock for Span, hashCode: 125992315)
);
-> at com.google.api.gax.tracing.OpencensusTracerFactoryTest.testImplicitParentSpan(OpencensusTracerFactoryTest.java:102)

However, there were exactly 2 interactions with this mock:
internalTracer.getCurrentSpan();
-> at com.google.api.gax.tracing.OpencensusTracerFactory.newTracer(OpencensusTracerFactory.java:101)

internalTracer.spanBuilderWithExplicitParent(
    "FakeClient.FakeMethod",
    null
);
-> at com.google.api.gax.tracing.OpencensusTracerFactory.newTracer(OpencensusTracerFactory.java:110)


Error:    OpencensusTracerTest.testLongRunningExample:152 
Argument(s) are different! Wanted:
span.addAnnotation(
    "Operation started",
    {}
);
-> at com.google.api.gax.tracing.OpencensusTracerTest.testLongRunningExample(OpencensusTracerTest.java:152)
Actual invocations have different arguments:
span.addAnnotation(
    "Scheduling next poll",
    {"delay ms" = AttributeValueLong{longValue=5}, "attempt" = AttributeValueLong{longValue=0}, "status" = AttributeValueString{stringValue=OK}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.attemptFailed(OpencensusTracer.java:353)
span.addAnnotation(
    "Operation started"
);
-> at com.google.api.gax.tracing.OpencensusTracer.lroStartSucceeded(OpencensusTracer.java:290)
span.addAnnotation(
    "Polling completed",
    {"attempt" = AttributeValueLong{longValue=1}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.attemptSucceeded(OpencensusTracer.java:323)
span.putAttributes(
    {"attempt count" = AttributeValueLong{longValue=2}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.operationSucceeded(OpencensusTracer.java:254)
span.end(
    
);
-> at com.google.api.gax.tracing.OpencensusTracer.operationSucceeded(OpencensusTracer.java:255)

[INFO] 
Error:  Tests run: 614, Failures: 2, Errors: 0, Skipped: 0

It seems I have somehow broke OpenCensus with this change.

@lqiu96 lqiu96 force-pushed the directpath-compatible-getter branch from 7e641ee to 5896193 Compare May 23, 2024 20:13
@lqiu96
Copy link
Contributor Author

lqiu96 commented May 23, 2024

Error:

Error:  Failures: 
Error:    OpencensusTracerFactoryTest.testImplicitParentSpan:102 
Wanted but not invoked:
internalTracer.spanBuilderWithExplicitParent(
    <any string>,
    same(Mock for Span, hashCode: 125992315)
);
-> at com.google.api.gax.tracing.OpencensusTracerFactoryTest.testImplicitParentSpan(OpencensusTracerFactoryTest.java:102)

However, there were exactly 2 interactions with this mock:
internalTracer.getCurrentSpan();
-> at com.google.api.gax.tracing.OpencensusTracerFactory.newTracer(OpencensusTracerFactory.java:101)

internalTracer.spanBuilderWithExplicitParent(
    "FakeClient.FakeMethod",
    null
);
-> at com.google.api.gax.tracing.OpencensusTracerFactory.newTracer(OpencensusTracerFactory.java:110)


Error:    OpencensusTracerTest.testLongRunningExample:152 
Argument(s) are different! Wanted:
span.addAnnotation(
    "Operation started",
    {}
);
-> at com.google.api.gax.tracing.OpencensusTracerTest.testLongRunningExample(OpencensusTracerTest.java:152)
Actual invocations have different arguments:
span.addAnnotation(
    "Scheduling next poll",
    {"delay ms" = AttributeValueLong{longValue=5}, "attempt" = AttributeValueLong{longValue=0}, "status" = AttributeValueString{stringValue=OK}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.attemptFailed(OpencensusTracer.java:353)
span.addAnnotation(
    "Operation started"
);
-> at com.google.api.gax.tracing.OpencensusTracer.lroStartSucceeded(OpencensusTracer.java:290)
span.addAnnotation(
    "Polling completed",
    {"attempt" = AttributeValueLong{longValue=1}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.attemptSucceeded(OpencensusTracer.java:323)
span.putAttributes(
    {"attempt count" = AttributeValueLong{longValue=2}}
);
-> at com.google.api.gax.tracing.OpencensusTracer.operationSucceeded(OpencensusTracer.java:254)
span.end(
    
);
-> at com.google.api.gax.tracing.OpencensusTracer.operationSucceeded(OpencensusTracer.java:255)

[INFO] 
Error:  Tests run: 614, Failures: 2, Errors: 0, Skipped: 0

It seems I have somehow broke OpenCensus with this change.

This seems to due to the added mockito-inline dependency used to mock static methods. I'm not sure why it's causing issues in non-related unit tests, so I'm going to remove this and look for a different way to test.

@lqiu96 lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label May 23, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label May 23, 2024
@lqiu96 lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label May 23, 2024
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label May 23, 2024
Comment on lines 88 to 97
static {
try {
systemProductName =
Files.asCharSource(new File("/sys/class/dmi/id/product_name"), StandardCharsets.UTF_8)
.readFirstLine();
} catch (IOException e) {
// Keep existing behavior the same (null means it is not on compute engine)
systemProductName = null;
}
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SystemProductName logic moved to a static block and initialized once. Stored in a variable so that it can be overriden.

Comment on lines 780 to 788
/**
* Package-Private scope as it is used to test DirectPath functionality in tests. This overrides
* the computed systemProductName when the class is initialized.
*/
@VisibleForTesting
Builder setSystemProductName(String systemProductName) {
this.systemProductName = systemProductName;
return this;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Package-Private setter to allow tests to override the computed SystemProductName

@lqiu96
Copy link
Contributor Author

lqiu96 commented May 24, 2024

Based on #2818, I'll need to remove junit-pioneer and add mocks for EnvProvider.

* <p>If productName is null, that represents the result of an IOException
*/
@VisibleForTesting
InstantiatingGrpcChannelProvider(Builder builder, String productName) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the EnvProvider as another param would need to make the envProvider variable non-final. I kept the package-private setter for now until we can migrate to using junit-pioneer to set the env vars.

.readFirstLine();
} catch (IOException e) {
// Keep existing behavior the same (null means it is not on compute engine)
systemProductName = null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can return an empty String so that we don't have to do a null check below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Updating.

* @return if DirectPath is enabled for the client AND if the configurations are valid
*/
@InternalApi
public boolean canUseDirectPath() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are conditions that we have canUseDirectPath is true but isDirectPathXdsEnabled is false based on the current logic, maybe we can expose isDirectPathXdsEnabled as a public method, and the Spanner team can set a client level attribute based on canUseDirectPath() && isDirectPathXdsEnabled()?

@lqiu96 lqiu96 requested a review from blakeli0 May 28, 2024 14:45
Copy link

sonarcloud bot commented May 28, 2024

Quality Gate Failed Quality Gate failed for 'gapic-generator-java-root'

Failed conditions
71.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

Copy link

sonarcloud bot commented May 28, 2024

Quality Gate Failed Quality Gate failed for 'java_showcase_integration_tests'

Failed conditions
41.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants