Document how to customize tags in micrometer-java11 HttpClient instrumentation

[claudemiro-oviva]

Currently my company is transitioning to the Quarkus ecosystem, we've found the integration with Micrometer nothing less than amazing. However, we must continue supporting applications that rely on other frameworks and various external SDKs. Our goal is to standardize metrics across these different libraries and frameworks.

Quarkus provides excellent metrics instrumentation for http_client_* using the microprofile-rest-client, as you can see the result below:

# HELP http_client_requests_seconds  
# TYPE http_client_requests_seconds summary
http_client_requests_seconds_count{clientName="stage.code.quarkus.io", method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 4.0
http_client_requests_seconds_sum{clientName="stage.code.quarkus.io", method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 0.862757042
# HELP http_client_requests_seconds_max  
# TYPE http_client_requests_seconds_max gauge
http_client_requests_seconds_max{clientName="stage.code.quarkus.io", method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 0.525936375

However, using micrometer-java11, the metrics are slightly different, lacking the clientName tag:

# HELP http_client_requests_seconds Timer for JDK's HttpClient
# TYPE http_client_requests_seconds summary
http_client_requests_seconds_count{method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 4.0
http_client_requests_seconds_sum{method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 1.301264958
# HELP http_client_requests_seconds_max Timer for JDK's HttpClient
# TYPE http_client_requests_seconds_max gauge
http_client_requests_seconds_max{method="GET", outcome="SUCCESS", status="200", uri="/extensions",} 0.570812834

Despite exploring the library, I have not found a way to customize micrometer-java11 to mirror the clientName tag functionality. I am aware of the uriMapper capability to customize the uri value.

Here are my questions:

1. What approach would you recommend for identifying the client or host in external requests?
2. Could custom metrics be a viable solution to this challenge?
3. Would you be interested in a pull request that introduces custom tags to micrometer-java11?

[shakuzen]

Thank you for opening the issue. How is the clientName tag derived in the metrics for the microprofile-rest-client?

For the HttpClient instrumentation in the micrometer-java11 module, it is based on the Observation API. To customize the tags on metrics, you can provide a custom ObservationConvention. You can extend the default one for easily adding a single keyvalue like the following:

class MyHttpClientObservationConvention extends DefaultHttpClientObservationConvention {
    @Override
    public KeyValues getLowCardinalityKeyValues(HttpClientContext context) {
        return super.getLowCardinalityKeyValues(context).and("clientName", deriveClientName(context));
    }
}

When creating the MicrometerHttpClient from micrometer-java11, you can pass it an instance of the above custom ObservationConvention.

We recently added basic documentation for the HttpClient instrumentation (currently available in the snapshot docs here). This is a good reminder to expand that documentation to cover use cases like this.

Does that answer your questions?

[claudemiro-oviva]

Hello @shakuzen,

Thanks for getting back to me. I tried your suggestion, but I'm still not seeing the custom tags. Also, I set breakpoints to check if the methods are being called, and it seems they aren't.

package com.example;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.java11.instrument.binder.jdk.MicrometerHttpClient;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.inject.Produces;
import jakarta.inject.Inject;
import java.net.http.HttpClient;

@ApplicationScoped
public class HttpClientProvider {
  @Inject MeterRegistry meterRegistry;

  @Produces
  HttpClient httpClient() {
    var httpClient = HttpClient.newBuilder().build();
    return MicrometerHttpClient.instrumentationBuilder(httpClient, meterRegistry)
        .customObservationConvention(new CustomDefaultHttpClientObservationConvention("theClient"))
        .uriMapper((request) -> request.uri().toString())
        .build();
  }
}

package com.example;

import io.micrometer.common.KeyValues;
import io.micrometer.common.lang.NonNull;
import io.micrometer.java11.instrument.binder.jdk.DefaultHttpClientObservationConvention;
import io.micrometer.java11.instrument.binder.jdk.HttpClientContext;

public class CustomDefaultHttpClientObservationConvention
    extends DefaultHttpClientObservationConvention {
  private final String clientName;

  public CustomDefaultHttpClientObservationConvention(String clientName) {
    this.clientName = clientName;
  }

  @Override
  @NonNull
  public KeyValues getLowCardinalityKeyValues(HttpClientContext context) {
    if (context.getCarrier() == null) {
      return KeyValues.empty();
    }
    return super.getLowCardinalityKeyValues(context).and("clientName", clientName);
  }
}

# HELP http_client_requests_seconds Timer for JDK's HttpClient
# TYPE http_client_requests_seconds summary
http_client_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="https://httpbin.org/status/200",} 16.0
http_client_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="https://httpbin.org/status/200",} 12.901033083
# HELP http_client_requests_seconds_max Timer for JDK's HttpClient
# TYPE http_client_requests_seconds_max gauge
http_client_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="https://httpbin.org/status/200",} 4.147540292

How is the clientName tag derived in the metrics for the microprofile-rest-client?

Answering your question, Quarkus uses the host to populate this value.
e.g:

http_client_requests_seconds_count{clientName="stage.code.quarkus.io",method="GET",outcome="SUCCESS",status="200",uri="/extensions",} 5.0

So, even if this works, it can also lead issues since the method getLowCardinalityKeyValues doesn't have access to the Http request.

[shakuzen]

Sorry, I should have mentioned, ObservationConvention will only be taken into account when instrumenting with the Observation API which requires configuring an ObservationRegistry. Without configuring that, instrumentation is done with only the MeterRegistry and then indeed there isn't a good way to customize tags as you found.

MicrometerHttpClient.instrumentationBuilder(httpClient, meterRegistry)
        .observationRegistry(observationRegistry) // use an ObservationRegistry for instrumentation
        .customObservationConvention(new CustomDefaultHttpClientObservationConvention("theClient"))
        .uriMapper((request) -> request.uri().toString())
        .build();

I'm not sure if Quarkus produces an ObservationRegistry for you to use for this or not. If that isn't available yet, you could make your own easily enough:

@Produces
ObservationRegistry observationRegistry() {
    ObservationRegistry registry = ObservationRegistry.create();
    registry.observationConfig().observationHandler(new DefaultMeterObservationHandler(meterRegistry));
    return registry;
}

The DefaultMeterObservationHandler is what will produce metrics from the Observation-based instrumentation.

the method getLowCardinalityKeyValues doesn't have access to the Http request.

The HTTP request is available from HttpClientContext#getCarrier. I didn't see a way to get the Host header from the HTTP request, but I guess if you aren't setting the Host yourself it is going to be derived from the URI and you could do the same yourself: context.getCarrier().build().uri().getHost().

[claudemiro-oviva]

Amazing @shakuzen, I was able to add the custom metrics. Thank you for your support!

[shakuzen]

I'm glad you were able to get it to work. We'll use this issue to track improving the documentation to mention configuring a custom ObservationConvention to customize the tags, then, unless you think there's any other change that needs to be made.

[claudemiro-oviva]

@shakuzen, I currently don't need this feature, but I suggest an improvement for the HttpClientContext. As of now, access is limited to using the Request builder, which could lead to a large number of objects that later need to be garbage collected. Additionally, there's no way to access the Response. For example, if we need to extract a header from the Response to create a tag, that isn't possible with the current setup. Ideally, access to both the Request and Response objects should be provided within the HttpClientContext. Please let me know if there are existing solutions I might have overlooked.

[shakuzen]

As of now, access is limited to using the Request builder, which could lead to a large number of objects that later need to be garbage collected.

As part of instrumentation, we may need to configure a header on the request, which is why we need the builder rather than an immutable request.

Additionally, there's no way to access the Response. For example, if we need to extract a header from the Response to create a tag, that isn't possible with the current setup.

There is a getResponse method available on HttpClientContext and you can see it used in the DefaultHttpClientObservationConvention. For example, the status and outcome tag values are derived from the response.

[claudemiro-oviva]

You are right @shakuzen. Thanks for your explanation.