2024-01-18 kernel meeting notes

[zachschuermann]

summary

API discussions, col batch discussion, delta 4.0, new RFC process incoming!

action items

from before the break (check on these- are they stale?)

• Nick design doc
• new hackathon/FFI code review
• kernel write draft review

attendees

@tdas @roeap @zachschuermann @nicklan @rtyler @ryan-johnson-databricks

notes stream

• data flow passing PR (see issue Using an Enum to pass data types #100)
• java world: everything can interop via ColumnarBatch API. Mix and match readers/expression handlers/etc.
• problem on rust side: we have a narrower interface and are relying on having an opaque data type that the engine can use whatever it wants and we provide some thin shims to get limited information from it.
• e.g. use MY (client's) custom parquet reader and want to interop with the default client's expression handler. HOW?
• TD: java side's heavier API worked since all existing engines use the same format/col vec APIs
• nick: note there is difference with native side since java has GC
• more to think about when there isn't GC. how do values propagate and get allocated/freed/iterated.
• robert: struct type with dataframe API that allows us to not care about a lot of this? problem is that the engine has to understand more detail (engine must know what the datatype really is). e.g. DuckDB has a custom parquet reader that reads to their own format but wants to integrate with our expression handler.
• two paths: (1) we have some API that allows for not forcing conversion to Arrow or (2) we just say "you must use arrow" to use default implementations (for DuckDB this wouldn't be a problem since they implement zero-copy conversions to arrow from their in-mem format)
• would we delineate between different kinds of schemas?
• ryan has thoughts (four thoughts to be exact)
  • (4) data skipping: our schema for skipping/file stats. need to do more investigation. most clients happy with default data skipping. If we want to enable customization we need to expose stats column. delta-sharing-rs wants to use kernel in client and server side for delta sharing. sharing: engine says here is name of table, partition, filters, and server gives back possibly pruned data. then we get back delta table (maybe with DVs maybe with filters etc.) but we apply it ourselves. you just get back single-commit delta table. lots of information hiding - pretty good sharing setup.
  • (3) native client vs jvm side (col batch things): memory management aspect and col batch aspect can be (partly) decoupled. two problems: (1) API mismatches - if the left duckDB native and right is arrow compute then they don't match. (2) when accessing these things (data) what is the memory story. rust side is ok (lifetimes) but FFI concerning.
  • (2) incompatibility: JVM side - everything in default client is defined in terms of col batch that allows get/set specific values. expression handler is basically a for loop now that just does the naive thing. rust kernel side: arrow as underlying format for default client. if anyone provides something other than arrow then we can't interop. solution: we COULD implement a for-loop scalar type thing like JVM. if we have both then clients get to pick.
  • (1) uh oh i missed one?
• robert: concern with for loops. context - arrow refactor of delta-rs. risks: arrow compute API could have slightly different semantics somehow. we need to be aware of this in general. no known gaps right now (timestamps might be weird?). if arrow-compute did manual SIMD better than rust compiler...
• tyler: delta wants to create Delta 4?
  • when spark 4 launches there might be delta 4? spark 4.0 -> spark connect on by default. want delta on this
  • does this put timeline on kernel work? unsure for now.
  • what is delta 4? see above (on spark 4.0 with spark connect)
  • will databricks announce kernel with delta 4? unsure for now.
• ryan: two things in 4.0: (1) match with spark versions - align and have congruent story and (2) target for summit. features still TBD
• tyler: importance of 4.0 - still having discussions on 1.0 in delta-rs side. care about API solidification not marketing. beneficial if we do 4.0/1.0 simultaneously. if we (delta-rs) does 1.0 we want to make sure of API compat with kernel efforts.
• ryan: suspicion - jvm kernel in 4.0. kernel rust uncertain timeline due to limited resourcing
• horray for learning abuot 4.0 >5mo before release.
• ryan: new delta features in ryan/bart's teams. consensus on having RFC process for changes to delta spec. yay! similar to rust rfc process maybe not as heavyweight.
• tyler: resourcing for DAT? limited. intern project?
• nick: let me (nick) know about intern projects!