Upgrade Kafka consumer to IBM/sarama driver

[yurishkuro]

According to #2461, Shopify/sarama is going to be sunset and IBM has started to maintain the library going forward: https://github.com/IBM/sarama.

The upgrade involves the following parts:

• Setup a testing pipeline and execute end to end tests that validate various edge cases that are currently accounted for in the consumer code, including
  • crash of a consumer and subsequent rebalancing of partitions
  • unavailability of the broker (crash or network partition) and resuming of work by the consumer when it's back
  • any other scenarios TBD
• Investigate the possibility of migrating to OpenTelemetry Kafka receiver and exporter, so that we do not have to maintain similar code. If possible skip the next item.
• Remove sarama-cluster dependency. This work has been largely done in Upgrade github.com/Shopify/sarama v1.33.0 to github.com/IBM/sarama v1.40.0 #4294, but should go through more rigorous testing described above.
• Upgrade to IBM/sarama (either our code or OTEL components, based on decision above)
• (bonus) Integrate testing scenarios into existing e2e test that runs in CI

[olad5]

Expression of Interest in LFX Mentorship - Productionize Streaming Jobs for Service Dependencies

Hello everyone,

I hope this message finds you well. I am thrilled to express my genuine interest in participating in the LFX mentorship program for the September to November term, specifically as part of the Jaeger Tracing project. The prospect of contributing to "Productionize Streaming Jobs for Service Dependencies" under the umbrella of Jaeger Tracing, an observability tool that has continually fascinated me, is both exciting and inspiring.

In the past weeks, I have worked on a projects that gave the basic understanding of observability in modern microservices architectures. One of the my projects involved building a simple microservices application that harnessed the power of Jaeger to efficiently monitor and analyze requests between two services, each supported by its own database (MongoDB and Postgres). This experience allowed me to appreciate the crucial role of observability in enhancing system performance and identifying potential issues proactively. Github Link: https://github.com/olad5/productive-pulse

Another project I built centered around the utilization of Kafka, with the Sarama Client . In this project, I designed a straightforward yet effective system that seamlessly processed student reports sourced from a CSV file, transforming and storing the results in a structured JSON format. This project, like the former, reinforced the significance of monitoring and streamlining data flows in a distributed environment. Github Link: https://github.com/olad5/kafka-sheet-processor

Driven by my passion for continuous learning, I have dedicated significant time to studying opentelemetry and Jaeger tracing. My explorations have led me to valuable resources such as:

https://opentelemetry.io/docs/instrumentation/go/getting-started/

https://medium.com/jaegertracing/jaeger-tracing-a-friendly-guide-for-beginners-7b53a4a568ca

https://www.aspecto.io/blog/opentelemetry-go-getting-started/
As I embark on this exciting mentorship opportunity, I am aware that accessing additional resources and guidance from the project maintainers will be invaluable in my quest to contribute effectively to the Jaeger Tracing project. With this in mind, I would like to politely request any relevant documentation, tutorials, or insights that could further enhance my understanding of the project's goals and technical aspects. Such assistance would undoubtedly empower me to deliver impactful contributions and foster meaningful collaboration within the Jaeger Tracing community.

I am genuinely excited about the possibility of being part of this mentorship program and actively contributing to the advancement of the Jaeger Project.

[yurishkuro]

@olad5 to be clear, neither streaming jobs nor this issue were selected as projects for the Fall round of LFX mentorship (#4457).

regarding the resources, you already found good ones, this one is also a deep walkthrough https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941

[olad5]

@yurishkuro , Thank you for responding. I now understand that the Elasticsearch Issue, will be the one selected for the mentorship. I also thank you for the resources you sent. I will look more into Elasticsearch because I have prior experience using zinc search https://github.com/zincsearch/zincsearch.

Again thank you for your time.

[slayer321]

Hey @yurishkuro , I did like to work on this issue. I was going through the following parts that you have mentioned and have some doubts.

• In the first part where you have mentioned about setup testing pipeline do you mean first we need to create the e2e test and then run it on CI/CD pipeline. Because currently I cannot see any e2e test on the project .(though on the last point i.e bonus one you have mentioned about e2e but I'm not sure which one you are referring to)
• I was going through the OTEL contrib code and as you mentioned that most of the code are similar.. I was comparing some of the fields from our auth dir and can see that some of them are not there on the OTEL contrib repo. I'm not sure how to go further with this any pointers on this will be helpful.
• I can see this PR is already raised but as it is not yet merged. So do we want to get this PR merged first or is it independent from this issue ?

[yurishkuro]

@slayer321 The PR #4294 is pretty complex, this is why I did not merge it, as there's no way to validate that it's working correctly aside from some basic sanity checks. We do have an e2e test for Kafka (plugin/storage/integration/kafka_test.go), but it only goes through the happy path, whereas the main complexity in the original code was from dealing with failure scenarios, rebalancing, etc.

We currently have an LFX mentorship project (#4600) to build ESv8 support, and the plan is to build it as an OTEL Exporter, since we have the desire to switch Jaeger collector onto OTEL collector framework. Once we do that, we could actually retire the Kafka-related code in Jaeger and use OTEL collector's exporter (it already supports the same formats and is already using IBM driver afaik). However, Jaeger code had specific features to support out of order processing of spans in the queue, I am not sure OTEL exporter has that.

It would still be nice to have the resilience testing implemented. Of course, one could argue that IBM driver should already provide that testing.

[slayer321]

Ok, for now I will work on some other issues.