Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation page for "big rocks" #76

Open
kylebarron opened this issue Jul 17, 2022 · 0 comments
Open

Documentation page for "big rocks" #76

kylebarron opened this issue Jul 17, 2022 · 0 comments
Labels
documentation Improvements or additions to documentation

Comments

@kylebarron
Copy link
Member

The idea is that we should document in a public place the big hurdles to making geopolars more performant and interoperable.

  • Polars does not support FixedSizeList... means that we can't today use the arrow-native geometry format defined in geo-arrow-spec
  • Polars does not support custom extension types... means that we can't consume data coming in from other applications that conforms to the latest draft of geo-arrow-spec that wraps geometry types like geoarrow.wkb in an ExtensionType("geoarrow.wkb", BinaryArray).
  • Polars does not use the arrow BinaryArray, but rather uses List<Uint8Array>. This means the recent geozero work to make reading faster from a BinaryArray is unusable unless it's also implemented for ListArray<u8>. This also means receiving data from other applications that store it as BinaryArray requires a cast.
  • Geometry traits in geo-types... means that we need at least two copies: one into the geo struct and another back into arrow memory after processing. We have a third right now that copies the arrow data into a Vec<u8>
@kylebarron kylebarron added the documentation Improvements or additions to documentation label Oct 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

1 participant