Delta support

[jbadeau]

Hi,

Is there any further progress on Delta support? If not then would using Raw DataFrameWriter/Reader be an option? This would make it non portable but would solve my immediate issues with using delta with beam.

Any pointers, tips?

Cheers

[iemejia]

Hi, I approached supporting delta by matching Parquet files from the Delta standalone reader. However I was hit by some dependency conflicts that I did not have the time to tackle. Another alternative would be just to 'write a thin' client for this. What matters for Beam are the links to the Parquet files and then you can read those via ParquetIO.readFiles.

If you are up to work on the task this is my WIP branch (I can help you with the review if so but I am too booked to work on this at the moment).
https://github.com/iemejia/beam/tree/deltaio

[jbadeau]

Hi,

Thank for the reply. I will try to get this going next week. Beam is pretty new to me (but I like it allot) so gonna need to dig into the api a bit.

What exactly do you mean by "thin-client"?

[iemejia]

Oversimplifying reading Delta is reading an index of which Parquet files contain the data + the associated schema(s). The current Delta standalone reader library that I used in the PR has some limitations but the biggest (from the Beam PoV) issue is that it leaks the json reading library (jackson) into the classpath which usually produces issues when running it with Beam, so one alternative would be to write a micro client just for reads in a way that did not leak classes.