Missing geo metadata in Parquet/Feather file which has a geo column

[marklit]

I'm using geopandas==0.14.2, pyarrow==15.0.0 and Python 3.11.7 on macOS 13.2.1. The Parquet file was built by DuckDB and is a normal Parquet file, not a GeoParquet file. There is a geo column in the Parquet file but GeoPandas is complaining that there isn't. Is it enough that the column is named geo or does there need to be some sort of special metadata in the PQ file before it could be read like this?

$ duckdb # 0.9.2

D DESCRIBE SELECT * FROM READ_PARQUET('flights.2023.11.19.pq') LIMIT 1;
┌─────────────┬─────────────────────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │                                   column_type                                   │  null   │   key   │ default │  extra  │
│   varchar   │                                     varchar                                     │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
│ dbFlags     │ BIGINT                                                                          │ YES     │         │         │         │
│ desc        │ VARCHAR                                                                         │ YES     │         │         │         │
│ icao        │ VARCHAR                                                                         │ YES     │         │         │         │
│ r           │ VARCHAR                                                                         │ YES     │         │         │         │
│ reg_details │ STRUCT(description VARCHAR, iso2 VARCHAR, iso3 VARCHAR, nation VARCHAR)         │ YES     │         │         │         │
│ t           │ VARCHAR                                                                         │ YES     │         │         │         │
│ timestamp   │ TIMESTAMP                                                                       │ YES     │         │         │         │
│ trace       │ STRUCT(aircraft STRUCT(alert BIGINT, alt_geom BIGINT, baro_rate BIGINT, categ…  │ YES     │         │         │         │
│ ownOp       │ VARCHAR                                                                         │ YES     │         │         │         │
│ year        │ BIGINT                                                                          │ YES     │         │         │         │
│ geo         │ BLOB                                                                            │ YES     │         │         │         │
├─────────────┴─────────────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┤
│ 11 rows                                                                                                                     6 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

D SELECT geo FROM READ_PARQUET('flights.2023.11.19.pq') LIMIT 1;
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                        geo                                                        │
│                                                       blob                                                        │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ \x00\x00\x18\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\xF9\xA3\xA83\xF7\x80 \xC0\x13\x9A$\x96\x94\x7F?@ │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

from geopandas import GeoDataFrame, read_parquet


df = read_parquet('flights.2023.11.19.pq',
                  columns=['geo',
                           'r',
                           'timestamp'],
                  filters=[('ownOp', '=', 'DELTA AIR LINES INC')])

File ~/.osm/lib/python3.11/site-packages/geopandas/io/arrow.py:407, in _arrow_to_geopandas(table, metadata)
    404 metadata = metadata or table.schema.metadata
    406 if metadata is None or b"geo" not in metadata:
--> 407     raise ValueError(
    408         """Missing geo metadata in Parquet/Feather file.
    409         Use pandas.read_parquet/read_feather() instead."""
    410     )
    412 try:
    413     metadata = _decode_metadata(metadata.get(b"geo", b""))

ValueError: Missing geo metadata in Parquet/Feather file.
            Use pandas.read_parquet/read_feather() instead.

[martinfleis]

GeoPandas.read_parquet is designed to read GeoParquet files. It is complaining that the file is not GeoParquet as there is no geo component of the file metadata (metadata != columns). If you have a non-geo Parquet with a geometry column, you should read it with pandas and then convert your geo column using from_wkb and upcast to GeoDataFrame.

does there need to be some sort of special metadata in the PQ file before it could be read like this?

Yes. See https://geoparquet.org/releases/v1.0.0/