KREST-7091 POC of request tracing with focus on Produce v3

[dimitarndimitrov]

This is a POC of a basic request tracing mechanism that intercepts requests, sets up a trace ID for each request, and traces (via logging) various stages of the request and response handling.

The POC uses Jersey filters for the main tracing pointcuts - it intercepts with a pre-matching request filter, post-matching request filter, and a response filter. The trace ID that can be extracted from the ContainerRequestContext can be used in the corresponding resources for further tracing (e.g. tracing just before making a Kafka RPC and just after receiving a result from it). The POC does that just for the Produce v3 API where it traces the parsing and sending of records, the reception of results (or errors), and the closing of the so-called produce batch or stream and the corresponding response channel.

It has been tested with a Produce v3 workload of ~80 concurrent req/sec and seemed to work well, being able to provide various data points about E2E latency from the Kafka REST POV, Produce v3 request outliers, etc.

[dimitarndimitrov]

I have pushed these changes to the main confluentinc repo of kafka-rest:

• https://github.com/confluentinc/kafka-rest/tree/KREST-7091 is a copy of the PR branch here
• https://github.com/confluentinc/kafka-rest/tree/KREST-7091-7.1.x adapts the changes to 7.1.x, where the Produce v3 code is noticeably different