Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't optimize a table created by Spark. #1648

Open
wjones127 opened this issue Sep 20, 2023 · 7 comments · May be fixed by #1650
Open

Can't optimize a table created by Spark. #1648

wjones127 opened this issue Sep 20, 2023 · 7 comments · May be fixed by #1650
Assignees
@rtyler
Labels
bug Something isn't working
Milestone
Correct timestamp...

Comments

@wjones127
Copy link
Collaborator

Environment

Delta-rs version:

Binding:

Environment:

  • Cloud provider:
  • OS:
  • Other:

Bug

What happened:

https://github.com/delta-io/delta-rs/actions/runs/6243973257/job/16950202833?pr=1602

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: NoMetadata', rust\src\operations\optimize.rs:1381:18

What you expected to happen:

How to reproduce it:

        #[tokio::test]
        async fn works_on_spark_table() {
            // Create a temporary directory
            let tmp_dir = tempdir::TempDir::new("optimize-spark").unwrap();
            let table_uri = tmp_dir.path().to_str().to_owned().unwrap();

            // Copy recursively from the test data directory to the temporary directory
            let source_path = "tests/data/delta-1.2.1-only-struct-stats";
            fs_extra::dir::copy(source_path, tmp_dir.path(), &Default::default()).unwrap();

            // Run optimize
            let (_, metrics) = DeltaOps::try_from_uri(table_uri)
                .await
                .unwrap()
                .optimize()
                .await
                .unwrap();

            // Verify it worked
            assert_eq!(metrics.num_files_added, 1);
        }

More details:
@wjones127 wjones127 added the bug Something isn't working label Sep 20, 2023
@wjones127 wjones127 mentioned this issue Sep 20, 2023
Closed
@rtyler
Copy link
Member

rtyler commented Sep 20, 2023

I saw this, I didn't realize it was a regression and thought it was related to the code in that pull request 🤔

@wjones127 are you going to be fixing this or is it "up for grabs"? 😄

@wjones127
Copy link
Collaborator Author

TBH I have no idea why this was in that PR. It doesn't have to do with upgrading PyArrow.

Up for grab for now, as I'll try to fix #1602 first.

@rtyler rtyler added this to the Rust v0.16 milestone Sep 20, 2023
@rtyler rtyler self-assigned this Sep 20, 2023
@rtyler
Copy link
Member

rtyler commented Sep 20, 2023

So the optimize actually fails but not for that reason. The reproduction case has to be modified slightly (which I have in my branch), after which the error is:

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: SchemaMismatch { msg: "Unexpected Arrow schema: g
ot: Field { name: \"integer\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"null\", data_type: Boolean, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Fi
eld { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"decimal\", data_type: Decimal128(8, 5), nulla
ble: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
 Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"date\", data_type: Date32, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata:
 {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { na
me: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metada
ta: {} }, Field { name: \"array\", data_type: List(Field { name: \"element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Struct([Field { n
ame: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, me
tadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", data_type: Struct([Field { name: \"struct_eleme
nt\", data_type: List(Field { name: \"element\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict
_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: fal
se, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", data_type: Int32, nullable: true, dict_id: 0,
 dict_is_ordered: false, metadata: {} }, expected: Field { name: \"integer\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field {
name: \"null\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_i
d: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name:
 \"decimal\", data_type: Decimal128(8, 5), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dic
t_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { nam
e: \"date\", data_type: Date32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Microsecond, None), nul
lable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { n
ame: \"entries\", data_type: Struct([Field { name: \"keys\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\",
data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"array\", data_type: List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"
struct_element\", data_type: Struct([Field { name: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable
: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", d
ata_type: Struct([Field { name: \"struct_element\", data_type: List(Field { name: \"item\", data_type: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"keys
\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", da
ta_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }" }', rust/src/operations/optimize.rs:1386:18

which looks much more like a real problem

@rtyler rtyler linked a pull request Sep 20, 2023 that will close this issue
Draft
@ion-elgreco
Copy link
Collaborator

ion-elgreco commented Sep 21, 2023 •
edited

So the optimize actually fails but not for that reason. The reproduction case has to be modified slightly (which I have in my branch), after which the error is:

thread 'operations::optimize::zorder::test::works_on_spark_table' panicked at 'called `Result::unwrap()` on an `Err` value: SchemaMismatch { msg: "Unexpected Arrow schema: g
ot: Field { name: \"integer\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"null\", data_type: Boolean, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Fi
eld { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"decimal\", data_type: Decimal128(8, 5), nulla
ble: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
 Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"date\", data_type: Date32, nullable: true,
dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Nanosecond, None), nullable: true, dict_id: 0, dict_is_ordered: false,
metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata:
 {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { na
me: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metada
ta: {} }, Field { name: \"array\", data_type: List(Field { name: \"element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Struct([Field { n
ame: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, me
tadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", data_type: Struct([Field { name: \"struct_eleme
nt\", data_type: List(Field { name: \"element\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict
_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: fal
se, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", data_type: Int32, nullable: true, dict_id: 0,
 dict_is_ordered: false, metadata: {} }, expected: Field { name: \"integer\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field {
name: \"null\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"boolean\", data_type: Boolean, nullable: true, dict_i
d: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"double\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name:
 \"decimal\", data_type: Decimal128(8, 5), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"string\", data_type: Utf8, nullable: true, dic
t_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"binary\", data_type: Binary, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { nam
e: \"date\", data_type: Date32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp\", data_type: Timestamp(Microsecond, None), nul
lable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct\", data_type: Struct([Field { name: \"struct_element\", data_type: Utf8, nullable: tr
ue, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"map\", data_type: Map(Field { n
ame: \"entries\", data_type: Struct([Field { name: \"keys\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\",
data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable:
 true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"array\", data_type: List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict
_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"nested_struct\", data_type: Struct([Field { name: \"
struct_element\", data_type: Struct([Field { name: \"nested_struct_element\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable
: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"struct_of_array_of_map\", d
ata_type: Struct([Field { name: \"struct_element\", data_type: List(Field { name: \"item\", data_type: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"keys
\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"values\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_orde
red: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"new_column\", da
ta_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }" }', rust/src/operations/optimize.rs:1386:18

which looks much more like a real problem

The arrow schema that is being written has nanoseconds timestamps while microseconds is expected in the delta schema. Did somewhere the default got changed in pyarrow to NS unit?

Also these errors could probably be easier to read if only the difference is shown instead of the full two schemas

@rtyler
Copy link
Member

rtyler commented Sep 21, 2023

I hadn't had a chance to look deeper into this, good eye @ion-elgreco ! That means this is actually a test case for #1286! The reporter of that issue will be so happy to see this being worked on!

@ion-elgreco
Copy link
Collaborator

I hadn't had a chance to look deeper into this, good eye @ion-elgreco ! That means this is actually a test case for #1286! The reporter of that issue will be so happy to see this being worked on!

Is there something similar on the rust side for this: https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetReadOptions.html#pyarrow.dataset.ParquetReadOptions.coerce_int96_timestamp_unit

This is at least what I used to read spark-delta tables using Python bindings in delta-rs

@ion-elgreco ion-elgreco mentioned this issue Sep 21, 2023
Open
@rtyler rtyler modified the milestones: Rust v0.16, Rust v0.17 Sep 27, 2023
@ion-elgreco
Copy link
Collaborator

ion-elgreco commented Oct 3, 2023 •
edited

@rtyler people can avoid this error by writing tables with this spark setting. The annoying default is INT96.. spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")

@rtyler rtyler mentioned this issue Oct 26, 2023
Open
rtyler added a commit to rtyler/delta-rs that referenced this issue Feb 3, 2024
@rtyler
fix: optimize tables written by Spark
2acb628 
Fixes delta-io#1648
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rtyler rtyler

Labels
bug Something isn't working
Projects
None yet
Milestone
Correct timestamp handling
Development

Successfully merging a pull request may close this issue.

fix: optimize Spark written tables rtyler/delta-rs
3 participants
@rtyler @wjones127 @ion-elgreco