You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Originally posted by lantse March 28, 2024
We are using mappartition function in Spark to transform data in batch. The data read()/write() for each record are in a user defined adapter class. Is there a way to let the spline spark agent to listen to the write() function so that this specific transformation would not be missing in the data lineage record?
In another word, how do we supplement or add to the data lineage record for the area that the agent cannot collect by default? Thanks.
The text was updated successfully, but these errors were encountered:
Discussed in #798
Originally posted by lantse March 28, 2024
We are using mappartition function in Spark to transform data in batch. The data read()/write() for each record are in a user defined adapter class. Is there a way to let the spline spark agent to
listen
to the write() function so that this specific transformation would not be missing in the data lineage record?In another word, how do we supplement or add to the data lineage record for the area that the agent cannot collect by default? Thanks.
The text was updated successfully, but these errors were encountered: