fixing pandas 1.6.0 and 2.0.0 breaking changes #580
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The |
jtilly
approved these changes
Oct 21, 2022
| @@ -1827,7 +1827,7 @@ def test_dataframe_std_errors(regression_data, categorical, split, fit_intercept | |||
| alpha=0, family="normal", fit_intercept=fit_intercept | |||
| ) | |||
| mdl.fit(X=X, y=y) | |||
| X_sm = pd.get_dummies(X) | |||
| X_sm = pd.get_dummies(X).to_numpy(dtype=float) | |||
| if fit_intercept: | |||
There was a problem hiding this comment.
get_dummies also accepts a dtype argument.
|
I agree that this change is problematic. It caused the canary build of statsmodels to also start failing. I think this change unfortunately is going to cause a lot of problems, and it should seriously be considered for rolling back. The |
|
Oops. I should have made most of my case on the pandas tracker. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the development version of pandas,
get_dummiesgenerates boolean data, and when used in arithmetic operations, this becomes a numpy array with dtype python object.Glum was actually not the problem. When using a pandas dataframe as inputs, tabmat takes care to cast everything to the right dtype. The problem was with the tests themselves where they would pandas dataframes with
.valuesto acces the underlying numpy array.Coincidentally, 2 days ago pandas released the 2.0.0 development version. Since we started seeing the problem before that, it seems to be happening with 1.6.0 and 2.0.0.
This PR fixes #575.
TODO: remove daily CI on push
Checklist
CHANGELOG.rstentry