Cannot query S3 paths containing whitespace

[andygrove]

Describe the bug

I cannot register a path containing a space.

create external table test2 stored as parquet location 's3://andygrove-benchmark-data/trip data/yellow_tripdata_2022-06.parquet';
ObjectStore(NotFound { path: "trip%20data/yellow_tripdata_2022-06.parquet"

It works if I change the path to not have spaces:

create external table test stored as parquet location 's3://andygrove-benchmark-data/trip_data/yellow_tripdata_2022-06.parquet';
0 rows in set. Query took 0.429 seconds.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[andygrove]

@tustvold Here is another potential object store bug that I found

[tustvold]

This is sadly intentional, https://docs.rs/object_store/latest/object_store/path/struct.Path.html#path-safety, I'm not really sure what can be done about this...

[andygrove]

We should add least add a check and throw an error? The S3 error response is somewhat obscure.

It will be a shame if DataFusion and Ballista cannot support querying certain public data sets like nyc-tlc because they have spaces in paths. Other query engines support it. One example: https://coiled.io/blog/nyc-taxi-parquet-csv-index-error/

I will plan on digging into this more soon to understand the issue better.

[tustvold]

We should add least add a check and throw an error?

Agreed, I would have expected DataFusion to refuse that query as the parquet location is not a valid URL. Possibly ListingTableUrl is escaping rather than parsing.

It will be a shame if DataFusion and Ballista cannot support querying certain public data sets

I welcome alternative suggestions for how to handle path escaping

[andygrove]

Perhaps some kind of user-provided config to override the default behavior? I will take a look this weekend.

[tustvold]

Actually looking into this, we don't disallow spaces in paths, I think this might be a DataFusion bug 🤔

[tustvold]

There is a somewhat related issue I ran into when playing with this, but you most definitely can create, read, etc... objects with paths containing spaces - #2800

[tustvold]

I think this was fixed by #2801