Closed
Description
See #32844
The MeterRegistryCustomizer isn't getting created when the dependent ApplicationTags bean is instantiated from a properties file.
It works if an ApplicationTags
bean is created via code.
This is probably an ordering issue, where WavefrontTracingAutoConfiguration
runs after WavefrontMetricsAutoConfiguration
.
Metadata
Metadata
Assignees
Labels
Type
Projects
Relationships
Development
No branches or pull requests
Activity
oppegard commentedon Nov 17, 2022
I forked the Observability for Boot 3.0 blog post to reproduce this issue: https://github.com/oppegard/observability-boot-blog-post. Both the client and server
application.properties
are populated withmanagement.wavefront.tracing
values to instantiate an ApplicationTags bean from WavefrontTracingAutoConfiguration.If you check the metric
my.observation
from the Client app, orjvm.gc.overhead
from both Client and Server, they don't have the tagsapplication
,service
,cluster
, orshard
.If an ApplicationTags bean is created via code in
ClientApplication
, then the tags do show on metrics emited from its WavefrontMeterRegistry.philwebb commentedon Nov 18, 2022
This issue actually highlighted that
ApplicationTags
are now used for both metrics and tracing. As such, they don't make much sense undermanagement.wavefront.tracing
so I've moved them up one level.oppegard commentedon Nov 18, 2022
@philwebb
ApplicationTags
should only be instantiated if WF distributed tracing is enabled (by including themicrometer-tracing-reporter-wavefront
dependency). WF distributed tracing has tags to identify a microservice within a broader application: application, service, cluster, shard. We only want those tags to be included on metrics if DT is enabled.The fix pushed at fb5cdbd has the unintended consequence of always instantiating an
ApplicationTags
bean and including them on metrics, even whenmicrometer-tracing-reporter-wavefront
is not included.oppegard commentedon Nov 18, 2022
Also, I think it makes more sense for the properties to live on
management.wavefront.tracing
since it applies only when DT is enabled (even though it affects metrics and spans – yeah, kinda confusing). I can go either way on that though.wilkinsona commentedon Nov 18, 2022
That worries me a bit from an upgrade perspective, although I'm not sufficiently knowledgable about Wavefront to know what the impact may be.
If a user's already using Wavefront for metrics and they want to start using it for tracing as well, what would be the impact of the tags changing on their metrics just because they've started to use tracing? Could it break existing dashboards?
philwebb commentedon Nov 18, 2022
When I was searching for usages I found one in
WavefrontTracingAutoConfiguration.wavefrontSpanHandler
and one inWavefrontMetricsExportAutoConfiguration.wavefrontApplicationTagsCustomizer
. That made me think that the tags were common to metrics and tracing.It does feel a little odd that setting up
ApplicationTags
for tracing has an impact on metrics.oppegard commentedon Nov 18, 2022
We've viewed the
ApplicationTags
as being additional tags on metrics. However, if their existing metrics have tags of application, service, cluster, or shard, then there will be a conflict when they start using distributed tracing (for DT to work, theApplicationTags
bean needs to take precence). This is the current behavior withwavefront-spring-boot
for Boot 2.x apps.We have existing out-of-the-box dashboards in Wavefront that will only function when spans and metrics having matching
ApplicationTags
. So if a user were to adopt distributed tracing down the line, they need to adapt their existing dashboard or makeApplicationTags
conform to their existing values.The tags are used for things like displaying JVM info for a service, as well as RED metrics: https://docs.wavefront.com/tracing_basics.html#examine-application-red-metrics-using-service-dashboard
philwebb commentedon Nov 18, 2022
Let me see if I can summarize to make sure we understand things correctly:
With
wavefront-spring-boot
a user could defineApplicationTags
either as an@Bean
or usingwavefront.application.*
properties. (at least that's my understanding from reading https://docs.wavefront.com/wavefront_springboot.html).When the
ApplicationTags
bean exists, it's adapted to micrometer metrics tags. This is done in WavefrontMetricsConfiguration.It looks like tags are also required if JVM metrics are being exported. They are also needed for the
WavefrontTracer
It looks to me like
wavefront-spring-boot
also always creates anApplicationTags
bean. This is done inWavefrontAutoConfiguration
. There's also aApplicationTagsBuilderCustomizer
interface that can be used if the user wants to customize the tags before they are built.It feels like Spring Boot 3.0 is doing something pretty similar. We're creating the
ApplicationTags
bean from properties if it doesn't exist and applying it to metrics and tracing.I wonder if we should move the properties we've added under
management.wavefront.application
. That way, someone migrating fromwavefront-spring-boot
could rename theirwavefront.application
properties?AFAICT the
ApplicationTags
bean should always be created.9 remaining items