Sending in OpenLineage Format via Kafka?

[giveitem]

Hello everyone,

New programmer here, just wondering if it's possible for Spline agent to send out Spark lineage information in OpenLineage format via Kafka?

I know right now we can send out OpenLineage format via HTTP, and Spline format via Kafka. My naive approach to this will be adding a sort of converter function to the Kafka consumer. Because I'm very new to all of this, I'm curious what will be the best way possible for the Spline agent to send out OpenLineage format via Kafka?

Thank you and really appreciate this piece of technology.

[cerveada]

Hi, there are two options in this case:

1) create new dispatcher inside Spline Agent that will send to Kafka

The dispatcher for openlineage over http is already available and spline lineage over kafka is also available so it's only about combining the code from those two

2) use http-kafka proxy to take an http messaes and insert them in kafka

This actually already exist. Take a look at the following project: https://github.com/AbsaOSS/spline-openlineage
The rest-proxy subproject works as a http-kafka proxy. It is not realeased though, so you must build the artifact yourself.
It also means additional application running somewhere.

[giveitem]

Perfect, thank you so much for the answer and the project.