DOC replace pandas with Polars in examples/gaussian_process/plot_gpr_co2.py #28804
Conversation
raisadz
force-pushed
the
polars_co2_example
branch
from
40e30a1
to
96a6ae9
Compare
raisadz
force-pushed
the
polars_co2_example
branch
from
96a6ae9
to
77bdbd8
Compare
raisadz
changed the title
| co2_data = co2.frame | ||
| co2_data["date"] = pd.to_datetime(co2_data[["year", "month", "day"]]) | ||
| co2_data = co2_data[["date", "co2"]].set_index("date") | ||
| co2_data = pl.DataFrame({col: co2.frame[col].to_numpy() for col in co2.frame.columns}) |
There was a problem hiding this comment.
why do we need a round trip here? pl.DataFrame(pd.DataFrame) doesn't work?
There was a problem hiding this comment.
Thank you for your review. Converting from/to pandas requires pyarrow as a dependency, and because scikit-learn doesn't have it, the following error will occur when trying to convert a pandas dataframe to Polars:
ModuleNotFoundError: pa.array requires 'pyarrow' module to be installedThere is an old related issue converting pl.to_pandas pola-rs/polars#3398
There was a problem hiding this comment.
Opened an issue on the polars side: pola-rs/polars#15845
There was a problem hiding this comment.
Seems like we can do:
| co2_data = pl.DataFrame({col: co2.frame[col].to_numpy() for col in co2.frame.columns}) | |
| co2_data = pl.from_dataframe(co2.frame) |
There was a problem hiding this comment.
It looks like pl.from_dataframe still needs pyarrow for older Polars versions and the checks fail for the minimum Polars version 0.19.12,
There was a problem hiding this comment.
Since we don't really depend on polars and it's only used in our CI for docs, I'd be in favor of moving the min version to something that supports this.
WDYT @ogrisel @thomasjpfan
There was a problem hiding this comment.
It seems that pl.from_dataframe works without pyarrow starting from Polars version 0.20.4. I can increase the minimum required Polars version from 0.19.12 to 0.20.4 if you prefer this option.
There was a problem hiding this comment.
It seems that pl.from_dataframe works without pyarrow starting from Polars version 0.20.4. I can increase the minimum required Polars version from 0.19.12 to 0.20.4 if you prefer this option.
I am okay with this option.
Use `pl.from_dataframe()` instead if converting to `numpy` Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>
…into polars_co2_example
…version from 0.19.12 to 0.20.23
There was a problem hiding this comment.
So this PR now uses pola-rs/polars#15933 and directly calls pl.DataFrame which doesn't require PyArrow anymore for the types in the input here.
However, that means minimum required version is the one released yesterday (28.04.2024). I'm personally okay with this. But I'd rather have another set of eyes on this.
Reference Issues/PRs
Related to #28341
What does this implement/fix? Explain your changes.
Replacing pandas with Polars allows to avoid checking pandas version when resampling monthly CO2.