Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REGR: using loc to add rows doesn't preserve dtypes #3125

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

m-richards
Copy link
Member

@m-richards m-richards commented Jan 5, 2024

Potential ideas for solving this. As implemented, _constructor_from_mgr stays nice and clean, but relies too heavily on the pandas internals for my liking (though in an ideal world pandas would expose some way to handle less straightforward dtype concatenation cases).

Edit: This kind of approach is not going to work because of this if else in compat compat:
https://github.com/pandas-dev/pandas/blob/612823e824805b97a2dbe258ba808dc572083d49/pandas/core/dtypes/concat.py#L128-L144
This block assumes to_concat shares a common dtype, and with this trick that's not true

I expect we'll end up using something along the lines of the update to _constructor_from_mgr which is commented out at the moment.

@jorisvandenbossche
Copy link
Member

Interesting! That's a nice workaround on our side. I will try to think tomorrow a bit more about how this ideally would be tackled on the pandas side. But on the shorter term, I think this common_dtype trick is an interesting avenue to explore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants