Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Treats Embedded Arrow Schema as Authoritative #1663

Closed
tustvold opened this issue May 6, 2022 · 0 comments · Fixed by #1682
Closed

Parquet Treats Embedded Arrow Schema as Authoritative #1663

tustvold opened this issue May 6, 2022 · 0 comments · Fixed by #1682
Assignees
@tustvold
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented May 6, 2022 •
edited

Describe the bug

As pointed out by @jorisvandenbossche on https://issues.apache.org/jira/browse/ARROW-16184 the embedded arrow schema should not be treated as authoritative for a parquet file, and instead should be used in a purely advisory capacity, for example, to aid in inferring list offsets, etc...

In particular

[It should be used as] a description of the original Arrow schema, and not for a description of what is in the Parquet file / for the Parquet schema.

This behavior appears to have been introduced by @carols10cents in apache/arrow#8354 and apache/arrow#8330, as a mechanism to round-trip types correctly. We should ensure the solution preserves this where applicable

To Reproduce

See #1459

Expected behavior

The embedded arrow schema should not be relied upon to be authoritative

Additional context

Working on as part of #1655
@tustvold tustvold added the bug label May 6, 2022
@tustvold tustvold self-assigned this May 6, 2022
@tustvold tustvold mentioned this issue May 7, 2022
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
c744337 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
4b98333 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
6eb932c 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)

Consistent handling of unsupported arrow types in parquet (apache#1666)
@tustvold tustvold mentioned this issue May 9, 2022
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 11, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
e2f12de 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
alamb pushed a commit that referenced this issue May 13, 2022
@tustvold
Fix Parquet Arrow Schema Inference (#1682)
5b154ea 
* Separate parquet -> arrow conversion logic (#1655)

Don't treat embedded arrow schema as authoritative (#1663)

Fix projection of nested parquet files (#1652) (#1654)

Fix schema inference for repeated fields (#1681)

Support reading alternative list representations from parquet (#1680)

* Add more tests

* Pass pointers by reference

* More docs

* Fix lint

* Review feedback

* Review feedback

* Fix test failures related to #1697
@alamb alamb added the parquet Changes to the parquet crate label May 26, 2022
@tustvold tustvold mentioned this issue Jun 1, 2022
Closed
@tustvold tustvold mentioned this issue Jun 23, 2022
Closed
@tustvold tustvold mentioned this issue Aug 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
bug parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Parquet Reader's Arrow Schema Inference tustvold/arrow-rs
2 participants
@alamb @tustvold