New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation for custom observation conventions is missing important hints #40158
Comments
@jonatan-ivanov Do you have any thoughts on this one? I'm not sure that anyone on the Boot team has the experience to suggest how these things could be done. I'm also not sure where any extended documentation should live if it's needed. |
Solutions 2) 3) and 4) all look similar to me:
The fact that you can use Servlet request attributes as a context map is irrelevant here and I think this information really belongs to the Micrometer documentation, if it's not already there. I'm closing this issue as a result. |
@bclozel, let me give you some concrete "running" code, so you can see the similarities, but also the differences. It took me some time to find a minimal approach, which seems to serve lots of cases and I think it is worth documenting it in some way:
Propagating...
Consuming...
|
I think if some data is missing from the
I might not understand this but I think they are the same: add data to the Looking at your code also tells me that you might want a new Observations for your business logic. Also, can't you do this instead: observation.lowCardinalityKeyValues("someKey", someValue); |
@bclozel @jonatan-ivanov thanks for your comments
No no, you might have gotten me wrong, I don't miss data in the context, I just look for consistent elegant way to access dynamic business data during the customization of the tags, which naturally means, that it has to be pushed to the observation context manually beforehand like you mentioned here:
Though I did not want to solve it via extra observation (thanks for the suggestion), because this would add the "ugly" overhead of wrapping all code places with
Yes, that is true, but where to add it resp. how to consume it differs (as already mentioned):
And exactly this I would like to see documented with code examples, which would save time researching through the whole monitoring topic.
No, this appears not possible as the business logic is mostly within the |
@hadjiski I think that when it comes to collecting information somewhere in the application and attaching it to an observation, you have 3 choices:
If the information is local to the current request and is a cross cutting concern, like a customer id - it makes sense to add it to some ThreadLocal and apply it globally to many observations using an observation filter. If the information is even more global and completely static for the lifetime of the application, there are easier ways to globally contribute that information to all observations. If this information is local to some business logic, then a dedicated observation probably makes sense. You shouldn't have to wrap all calls like In all cases, I think that adding business data to server request attributes and using it as a carrier, out of band, to collect information for other observations, is not a great idea. It indeed looks like a missing custom observation in the application. At least this is how it looks with the information you've provided.
Yes, but Observation handlers will produce timers and traces. Counters and gauges don't really make sense with the I don't think we can improve the documentation as the issue is not really about learning how to do something, but choosing a solution to collect the relevant information and thinking about the tradeoffs. |
@hadjiski .. I have integrated my spring boot application with cassandra observability. Have configured it and able to push low cardinality metrices to prometheus. But unable to push high cardinality keys to prometheus with the existing code base. Can you please let me know which class need to be overridden and how to allow db.statement into low cardinality ? This will be helpful if you can please suggest on this |
@divyagangigunta123 this is by design. High cardinality key values should not be published with metrics as metrics systems are not meant to collect unbounded values. Please check the Micrometer docs for further information. |
@divyagangigunta123 On top of what Brian mentioned, you can read about this more and why this is a problem here: https://develotters.com/posts/high-cardinality/ |
Considering a standard spring boot rest controller application with some business and data repository layers + outgoing client communication:
Customizing observation conventions might get tricky, when you go beyond the trivial well documented examples in the spring boot docu, spring framework docu and micrometer
Here my thoughts:
ServerRequestObservationContext
->getCarrier()
ClientRequestObservationContext
mapScheduledTaskObservationContext
map(as you can see, for the "top" 4 metrics, we have 3 different ways of propagating business data)
I know people promote using of context propagation lib, which I did not study so far deeply, but it seems like an overhead for the small thing we are trying to achieve here. Not to mention the fact, that without the extra lib, we can rely on the context being reset for us, not worrying about clearing all the thread locals properly
It would be nice to align those, but I assume it is in the nature of things, when many projects are building on top of each other - then let it be better documented with examples
The text was updated successfully, but these errors were encountered: