You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you check the metric my.observation from the Client app, or jvm.gc.overhead from both Client and Server, they don't have the tags application, service, cluster, or shard.
If an ApplicationTags bean is created via code in ClientApplication, then the tags do show on metrics emited from its WavefrontMeterRegistry.
This issue actually highlighted that ApplicationTags are now used for both metrics and tracing. As such, they don't make much sense under management.wavefront.tracing so I've moved them up one level.
@philwebbApplicationTags should only be instantiated if WF distributed tracing is enabled (by including the micrometer-tracing-reporter-wavefront dependency). WF distributed tracing has tags to identify a microservice within a broader application: application, service, cluster, shard. We only want those tags to be included on metrics if DT is enabled.
The fix pushed at fb5cdbd has the unintended consequence of always instantiating an ApplicationTags bean and including them on metrics, even when micrometer-tracing-reporter-wavefront is not included.
Also, I think it makes more sense for the properties to live on management.wavefront.tracing since it applies only when DT is enabled (even though it affects metrics and spans – yeah, kinda confusing). I can go either way on that though.
That worries me a bit from an upgrade perspective, although I'm not sufficiently knowledgable about Wavefront to know what the impact may be.
If a user's already using Wavefront for metrics and they want to start using it for tracing as well, what would be the impact of the tags changing on their metrics just because they've started to use tracing? Could it break existing dashboards?
When I was searching for usages I found one in WavefrontTracingAutoConfiguration.wavefrontSpanHandler and one in WavefrontMetricsExportAutoConfiguration.wavefrontApplicationTagsCustomizer. That made me think that the tags were common to metrics and tracing.
It does feel a little odd that setting up ApplicationTags for tracing has an impact on metrics.
We've viewed the ApplicationTags as being additional tags on metrics. However, if their existing metrics have tags of application, service, cluster, or shard, then there will be a conflict when they start using distributed tracing (for DT to work, the ApplicationTags bean needs to take precence). This is the current behavior with wavefront-spring-boot for Boot 2.x apps.
We have existing out-of-the-box dashboards in Wavefront that will only function when spans and metrics having matching ApplicationTags. So if a user were to adopt distributed tracing down the line, they need to adapt their existing dashboard or make ApplicationTags conform to their existing values.
Let me see if I can summarize to make sure we understand things correctly:
With wavefront-spring-boot a user could define ApplicationTags either as an @Bean or using wavefront.application.* properties. (at least that's my understanding from reading https://docs.wavefront.com/wavefront_springboot.html).
When the ApplicationTags bean exists, it's adapted to micrometer metrics tags. This is done in WavefrontMetricsConfiguration.
It looks to me like wavefront-spring-boot also always creates an ApplicationTags bean. This is done in WavefrontAutoConfiguration. There's also a ApplicationTagsBuilderCustomizer interface that can be used if the user wants to customize the tags before they are built.
It feels like Spring Boot 3.0 is doing something pretty similar. We're creating the ApplicationTags bean from properties if it doesn't exist and applying it to metrics and tracing.
I wonder if we should move the properties we've added under management.wavefront.application. That way, someone migrating from wavefront-spring-boot could rename their wavefront.application properties?
AFAICT the ApplicationTags bean should always be created.
Activity
oppegard commentedon Nov 17, 2022
I forked the Observability for Boot 3.0 blog post to reproduce this issue: https://github.com/oppegard/observability-boot-blog-post. Both the client and server
application.properties
are populated withmanagement.wavefront.tracing
values to instantiate an ApplicationTags bean from WavefrontTracingAutoConfiguration.If you check the metric
my.observation
from the Client app, orjvm.gc.overhead
from both Client and Server, they don't have the tagsapplication
,service
,cluster
, orshard
.If an ApplicationTags bean is created via code in
ClientApplication
, then the tags do show on metrics emited from its WavefrontMeterRegistry.philwebb commentedon Nov 18, 2022
This issue actually highlighted that
ApplicationTags
are now used for both metrics and tracing. As such, they don't make much sense undermanagement.wavefront.tracing
so I've moved them up one level.oppegard commentedon Nov 18, 2022
@philwebb
ApplicationTags
should only be instantiated if WF distributed tracing is enabled (by including themicrometer-tracing-reporter-wavefront
dependency). WF distributed tracing has tags to identify a microservice within a broader application: application, service, cluster, shard. We only want those tags to be included on metrics if DT is enabled.The fix pushed at fb5cdbd has the unintended consequence of always instantiating an
ApplicationTags
bean and including them on metrics, even whenmicrometer-tracing-reporter-wavefront
is not included.oppegard commentedon Nov 18, 2022
Also, I think it makes more sense for the properties to live on
management.wavefront.tracing
since it applies only when DT is enabled (even though it affects metrics and spans – yeah, kinda confusing). I can go either way on that though.wilkinsona commentedon Nov 18, 2022
That worries me a bit from an upgrade perspective, although I'm not sufficiently knowledgable about Wavefront to know what the impact may be.
If a user's already using Wavefront for metrics and they want to start using it for tracing as well, what would be the impact of the tags changing on their metrics just because they've started to use tracing? Could it break existing dashboards?
philwebb commentedon Nov 18, 2022
When I was searching for usages I found one in
WavefrontTracingAutoConfiguration.wavefrontSpanHandler
and one inWavefrontMetricsExportAutoConfiguration.wavefrontApplicationTagsCustomizer
. That made me think that the tags were common to metrics and tracing.It does feel a little odd that setting up
ApplicationTags
for tracing has an impact on metrics.oppegard commentedon Nov 18, 2022
We've viewed the
ApplicationTags
as being additional tags on metrics. However, if their existing metrics have tags of application, service, cluster, or shard, then there will be a conflict when they start using distributed tracing (for DT to work, theApplicationTags
bean needs to take precence). This is the current behavior withwavefront-spring-boot
for Boot 2.x apps.We have existing out-of-the-box dashboards in Wavefront that will only function when spans and metrics having matching
ApplicationTags
. So if a user were to adopt distributed tracing down the line, they need to adapt their existing dashboard or makeApplicationTags
conform to their existing values.The tags are used for things like displaying JVM info for a service, as well as RED metrics: https://docs.wavefront.com/tracing_basics.html#examine-application-red-metrics-using-service-dashboard
philwebb commentedon Nov 18, 2022
Let me see if I can summarize to make sure we understand things correctly:
With
wavefront-spring-boot
a user could defineApplicationTags
either as an@Bean
or usingwavefront.application.*
properties. (at least that's my understanding from reading https://docs.wavefront.com/wavefront_springboot.html).When the
ApplicationTags
bean exists, it's adapted to micrometer metrics tags. This is done in WavefrontMetricsConfiguration.It looks like tags are also required if JVM metrics are being exported. They are also needed for the
WavefrontTracer
It looks to me like
wavefront-spring-boot
also always creates anApplicationTags
bean. This is done inWavefrontAutoConfiguration
. There's also aApplicationTagsBuilderCustomizer
interface that can be used if the user wants to customize the tags before they are built.It feels like Spring Boot 3.0 is doing something pretty similar. We're creating the
ApplicationTags
bean from properties if it doesn't exist and applying it to metrics and tracing.I wonder if we should move the properties we've added under
management.wavefront.application
. That way, someone migrating fromwavefront-spring-boot
could rename theirwavefront.application
properties?AFAICT the
ApplicationTags
bean should always be created.9 remaining items