Monitoring platform events with zipkin-go and opentracing #50
Conversation
Perf events can provide additional insights into the performance of a span or a sequence of spans in a trace. This patch includes the use of a library called "perfevents" (https://github.com/opentracing-contrib/perfevents) which can be simply imported and used. This library is an abstraction of the platform side of counters and operations. Right now, only the generic perf events are supported. With this patch, one should be able to find out the cpu-cycles, instructions, cache-references, cache-misses, branch-instructions, branch-misses and bus cycles for a span. The required event(s) should be supplied in the form of a string(comma separated list if there are multiple events to be monitored) to the StartSpan call and then, the opentracing-go library and the zipkin-go-opentracing library should take over from there. As soon as the span is closed, the perf event(s) count is logged with span.Log(). An example to use this with a span : // start a span with perfevents sp = tracer.StartSpan("read_file", opentracing.PerfString("cache-misses,cpu-cycles")) Upon closing the span, the events stop monitoring and are destroyed afer the data are sent for logging. Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>
|
opentracing/spec Issue in refernce : opentracing/specification#36 Also, the travis error mentioned above is because the perfevents package is not yet upstream : |
Use PerfEvent tag to check if perfevents needs to be initialized, if yes, initialize them. Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>
|
Added one more commit to this PR to use tags instead of span options to start the perfevents. |
|
I like the idea of being able to have these metrics inside spans, but not in the proposed current implementation where it breaks cross platform compatibility. I think most will depend on the outcome of: opentracing/opentracing-go#136 |
|
Hi @basvanbeek |
|
Hi, |
|
@hkshaw1990 unfortunately I have too much on my plate at the moment to look at an Observer interface more closely but I definitely don't dismiss the idea of one existing. In the mean time you could investigate a POC and adjust the PR to include this logic through an Observer interface and make it easier for all of us to look at the implications for the project as a whole. Since I consider the platform events logic as something for consumers to opt-in on, I would like the implementation of the Observer interface be neither obtrusive nor performance impeding. |
Metrics are useful to gain insights in a distributed application. But there can be a lot of metrics in different domains. Adding such metrics from one domain (metrics exporter pkg) into zipkin is not good for cross platform compatibility. This commit adds a new observer and span observer API which defines a standard interface to add such metrics. To expose the metrics, one would have to implement the observer interface in the metrics exporter. zipkin will need to implement this interface and install callbacks for methods such as StartSpan, SetOperationName, SetTag and Finish. The registered callbacks would then be called on the span events if an observer is created. This is based on the work done by Yuri here : jaegertracing/jaeger-client-go#94 Signed-off-by: Yuri Shkuro <yurishkuro@users.noreply.github.com> Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>
|
@basvanbeek I have added a commit as a POC to add an API for the Observer interface and its implementation. |
observer.go
Outdated
| // observer := myobserver.NewObserver() | ||
| // tracer := zipkin.NewTracer(..., zipkin.WithObserver(observer)) | ||
| // | ||
| type Observer interface { |
There was a problem hiding this comment.
Please remove the Observer interface here and use the one you created in https://github.com/opentracing-contrib/go-observer. You can pull PR opentracing-contrib/go-observer#2 to fix the missing dependencies to opentracing-go
There was a problem hiding this comment.
WIll do and thanks for the PR!
observer.go
Outdated
| // other Span events. | ||
| // zipkin should define these functions for each of the span operations | ||
| // which should call the registered (observer) callbacks. | ||
| type SpanObserver interface { |
There was a problem hiding this comment.
Same as Observer. This one can be removed
observer.go
Outdated
|
|
||
| // noopSpanObserver is used when there are no observers registered on the | ||
| // Tracer or none of them returns span observers | ||
| var noopSpanObserver = spanObserver{} |
There was a problem hiding this comment.
I added a NoopSpanObserver implementation to opentracing-contrib/go-observer.
If you use that one we don't try to range over SpanObservers when calling one of the SpanObserver methods
EDIT: Yuri told me it is fine to return nil, isn't that a much better idea?
There was a problem hiding this comment.
Yeah, makes sense to return a nil.
span.go
Outdated
| @@ -29,6 +29,7 @@ type Span interface { | |||
| type spanImpl struct { | |||
| tracer *tracerImpl | |||
| event func(SpanEvent) | |||
| Observer SpanObserver | |||
There was a problem hiding this comment.
do not export the SpanObserver variable as it is not needed and direct external access to it is undesirable.
There was a problem hiding this comment.
Ok, will change this.
|
@basvanbeek added the changes which uses the go-observer interfaces and addresses the review comments. |
|
Thanks @hkshaw1990 for your patience with this. |
Perf events can provide additional insights into the performance of a
span or a sequence of spans in a trace. This patch includes the use of
a library called
"perfevents" (https://github.com/opentracing-contrib/perfevents) which
can be simply imported and used. This library is an abstraction of the
platform side of counters and operations. Right now, only the generic
perf events are supported. With this patch, one should be able to find
out the cpu-cycles, instructions, cache-references, cache-misses,
branch-instructions, branch-misses and bus cycles for a span.
The required event(s) should be supplied in the form of a string(comma
separated list if there are multiple events to be monitored) to the
StartSpan call and then, the opentracing-go library and the
zipkin-go-opentracing library should take over from there. As soon as
the span is closed, the perf event(s) count is logged with span.Log().
An example to use this with a span :
// start a span with perfevents
sp = tracer.StartSpan("read_file",
opentracing.PerfString("cache-misses,cpu-cycles"))
Upon closing the span, the events stop monitoring and are destroyed afer
the data is sent for logging.