Lineage from Spark Event Logs

[Akash1125]

Hi All,

I would like to know if it is possible to

1. Derive Table and column level lineage from Spark Event logs (spark.eventLog.enabled=true) which are persisted (spark.eventLog.dir=). This means that a custom listener like Spline would not need to be registered in run time and we would not need to think about things like memory footprint of the listener etc. I looked at the event logs and see that "SparkListenerSQLExecutionStart" event seems promising as it contains the Physical/Logical plan info. Please let me know if I am thinking in the right direction.
   

2. If point 1 is possible, Would I be able to use Spline in some fashion (without registering the listener part) to gather the Lineage from persisted Event logs ? ( For example- in a batch job)

Please suggest.

Thanks.

[cerveada]

Some lineage information seems to be in the Event logs, but I am not sure if there is enough for every possible lineage.

Assuming all the lineage data are in the event log, you would have to write another agent (something like SparkEventLogParsingAgent), that would parse the lineage data and send it to the Spline server. Rest of the Spline (Server + UI) can then be used.