Add dataframe.nullable_dtypes configuration option
#9874
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jrbourbeau
commented
Jan 25, 2023
Comment on lines
+101
to
+106
| class _NoDefault(Enum): | ||
| no_default = ... | ||
|
|
||
|
|
||
| no_default: Final = _NoDefault.no_default | ||
| NoDefault = Literal[_NoDefault.no_default] |
There was a problem hiding this comment.
This change was needed to make mypy happy with the new default value of use_nullable_dtypes= in read_parquet. cc @crusaderky in case you think there's a better way to do this
There was a problem hiding this comment.
This is indeed the canonical way of doing it (and yes it's ugly)
|
Revisiting the example in #9631, we can now use just configuration options to get all In [1]: import dask
In [2]: import dask.dataframe as dd
In [3]: dask.config.set({"dataframe.dtype_backend": "pyarrow", "dataframe.nullable_dtypes": True})
Out[3]: <dask.config.set at 0x102dadb70>
In [4]: df = dd.read_parquet("s3://nyc-tlc/trip data/fhvhv_tripdata_2022-06.parquet", split_row_groups=True)
In [5]: df.dtypes
Out[5]:
hvfhs_license_num string[pyarrow]
dispatching_base_num string[pyarrow]
originating_base_num string[pyarrow]
request_datetime timestamp[us][pyarrow]
on_scene_datetime timestamp[us][pyarrow]
pickup_datetime timestamp[us][pyarrow]
dropoff_datetime timestamp[us][pyarrow]
PULocationID int64[pyarrow]
DOLocationID int64[pyarrow]
trip_miles double[pyarrow]
trip_time int64[pyarrow]
base_passenger_fare double[pyarrow]
tolls double[pyarrow]
bcf double[pyarrow]
sales_tax double[pyarrow]
congestion_surcharge double[pyarrow]
airport_fee double[pyarrow]
tips double[pyarrow]
driver_pay double[pyarrow]
shared_request_flag string[pyarrow]
shared_match_flag string[pyarrow]
access_a_ride_flag string[pyarrow]
wav_request_flag string[pyarrow]
wav_match_flag string[pyarrow]
dtype: object |
Awesome! Should also make switching the default easier at some point in the future. |
4 tasks
github-actions
bot
added
the
needs attention
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new
dataframe.nullable_dtypeconfiguration option similar to whatpandasis doing in pandas-dev/pandas#50748. This should both make alignment betweenpandasanddask.dataframebetter and make it easier for folks to opt into usingpyarrow-backed dtypes.cc @phofl for visibility