The road to deltalake v1.0.0 🦀

[rtyler]

I have been thinking about what API stability and what a celebration-worthy 1.0 release for the deltalake crate would entail. I think we're making great progress and have largely kept strong APIs over the past year or so. When I think about 1.0, the things I would like to see (and maybe implement) would be:

High level features:

• Change Data Feed support Support CDC #604 feat: introduce CDC write-side support for the Update operations #2486
• Constraints
• Schema evolution for all Rust/Python writers Schema evolution on upsert (merge) #2282
• Deletion vector support Support deletion vector #1094
• Rust-based writer engine for Python by default milestone #10
• Stablized API for DataFusion integration
• Stabilized API for configuration via environment variables for object store

High level bug fixes:

• Full timestamp support (INT96/Timestamp/Timestamp with Timezone), currently a persistent thorn in the side of multi-reader/writer setups.
• Adoption of AWS supported Rust SDKs for dependencies.

Depending on progress made by some projects driven by our friends at Databricks, I think it would also be exciting to incluhde:

• Preliminary delta-kernel-rs adoption, which would hopefully add deletion vector support. feat: adopt kernel schema types #2495
• Liquid clustering support, pending completion of the design doc. (Currently as written it just looks like it's a minimal scope ZORDER operation executed on writes, which we have the pieces for already.)
• Unity catalog integration pending credential vending and Unity supporting the APIs necessary for 3rd parties to interoperate.

I think that's "close enough" to be impement-able by 1.0 around Summit.

I wanted to share this with everybody to get a sense of what you think we need for a 1.0 release?

[ion-elgreco]

I only have one remark, if I remember correctly, Will mentioned that deletion vectors would require a new kind of reader on the python side. So am I not sure if that should be part of v1.0.0, since it could potentially take some time. Maybe deletion vectors can be v1.1

Also I started looking into the timestamp tz issue, it's not a huge change but it will have a breaking effect on current writers, so we should definitely do that sooner than later.

[rtyler]

@ion-elgreco My suggestion here is for the Rust library. I don't think we should hold Deletion Vectors back from the Rust implementation on account of more work being needed by the Python library 😄

I think that's a good call out for us to keep in mind with reader and writer changes in 🦀 since there are implications for 🐍

[jordandakota]

Is there somewhere to read about the concerns around supporting deletion vectors on the python side (obv rust needs it first)? We've found a lot of support for pulling spark out of a lot of our deltalake workloads since merge was introduced into delta-rs. With deletion vectors going enabled by default in most databricks workspaces, more external tables are being set up with deletion vectors which reduces the reach of delta-rs for us.

[rtyler]

Is there somewhere to read about the concerns around supporting deletion vectors on the python side (obv rust needs it first)? We've found a lot of support for pulling spark out of a lot of our deltalake workloads since merge was introduced into delta-rs. With deletion vectors going enabled by default in most databricks workspaces, more external tables are being set up with deletion vectors which reduces the reach of delta-rs for us.

@jordandakota There's #1094 which is kind of the placeholder for this. I know that @zachschuermann wrote some Deletion Vector support in the early days of the delta-kernel-rs work, but what is missing is the bringing of that work into delta-rs either through kernel (which @roeap is starting on #2489) or bringing the code over.

If you're open to rolling up your sleeves, that would be a great contribution to the project 😉

[yeshsriram]

Hope this #2006 makes it.