Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pivot_table with nested elements and numpy 1.24 #50682

Merged
merged 16 commits into from Jan 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.5.3.rst
Expand Up @@ -32,8 +32,8 @@ Bug fixes
- Bug in :meth:`Series.quantile` emitting warning from NumPy when :class:`Series` has only ``NA`` values (:issue:`50681`)
- Bug when chaining several :meth:`.Styler.concat` calls, only the last styler was concatenated (:issue:`49207`)
- Fixed bug when instantiating a :class:`DataFrame` subclass inheriting from ``typing.Generic`` that triggered a ``UserWarning`` on python 3.11 (:issue:`49649`)
- Bug in :func:`pivot_table` with NumPy 1.24 or greater when the :class:`DataFrame` columns has nested elements (:issue:`50342`)
- Bug in :func:`pandas.testing.assert_series_equal` (and equivalent ``assert_`` functions) when having nested data and using numpy >= 1.25 (:issue:`50360`)
-

.. ---------------------------------------------------------------------------
.. _whatsnew_153.other:
Expand Down
13 changes: 12 additions & 1 deletion pandas/core/common.py
Expand Up @@ -25,6 +25,7 @@
cast,
overload,
)
import warnings

import numpy as np

Expand Down Expand Up @@ -235,7 +236,17 @@ def asarray_tuplesafe(values: Iterable, dtype: NpDtype | None = None) -> ArrayLi
if isinstance(values, list) and dtype in [np.object_, object]:
return construct_1d_object_array_from_listlike(values)

result = np.asarray(values, dtype=dtype)
try:
with warnings.catch_warnings():
# Can remove warning filter once NumPy 1.24 is min version
warnings.simplefilter("ignore", np.VisibleDeprecationWarning)
result = np.asarray(values, dtype=dtype)
except ValueError:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I think it's better to just wrap the line that can raise in the try block. Is it a problem with the warning catching in this case? No big deal, I guess the warnings stuff won't raise a ValueError, but if it does, the behavior won't be as expected.

Copy link
Member

@phofl phofl Jan 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm in this case I think this is better as is, if we would wrap try-except into the catch_warnings statement, then we would also catch warnings in the except block, which isn't what we want here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah result = np.asarray(values, dtype=dtype) will raise a warning (due to our usage) w/ numpy < 1.24 and raise an exception with numpy >= 1.24. As mentioned, I don't want to accidentally mask a warning within the except block.

# Using try/except since it's more performant than checking is_list_like
# over each element
# error: Argument 1 to "construct_1d_object_array_from_listlike"
# has incompatible type "Iterable[Any]"; expected "Sized"
return construct_1d_object_array_from_listlike(values) # type: ignore[arg-type]

if issubclass(result.dtype.type, str):
result = np.asarray(values, dtype=object)
Expand Down
69 changes: 69 additions & 0 deletions pandas/tests/reshape/test_pivot.py
Expand Up @@ -2312,6 +2312,75 @@ def test_pivot_table_datetime_warning(self):
)
tm.assert_frame_equal(result, expected)

def test_pivot_table_with_mixed_nested_tuples(self, using_array_manager):
# GH 50342
df = DataFrame(
{
"A": ["foo", "foo", "foo", "foo", "foo", "bar", "bar", "bar", "bar"],
"B": ["one", "one", "one", "two", "two", "one", "one", "two", "two"],
"C": [
"small",
"large",
"large",
"small",
"small",
"large",
"small",
"small",
"large",
],
"D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
"E": [2, 4, 5, 5, 6, 6, 8, 9, 9],
("col5",): [
"foo",
"foo",
"foo",
"foo",
"foo",
"bar",
"bar",
"bar",
"bar",
],
("col6", 6): [
"one",
"one",
"one",
"two",
"two",
"one",
"one",
"two",
"two",
],
(7, "seven"): [
"small",
"large",
"large",
"small",
"small",
"large",
"small",
"small",
"large",
],
}
)
result = pivot_table(
df, values="D", index=["A", "B"], columns=[(7, "seven")], aggfunc=np.sum
)
expected = DataFrame(
[[4.0, 5.0], [7.0, 6.0], [4.0, 1.0], [np.nan, 6.0]],
columns=Index(["large", "small"], name=(7, "seven")),
index=MultiIndex.from_arrays(
[["bar", "bar", "foo", "foo"], ["one", "two"] * 2], names=["A", "B"]
),
)
if using_array_manager:
# INFO(ArrayManager) column without NaNs can preserve int dtype
expected["small"] = expected["small"].astype("int64")
tm.assert_frame_equal(result, expected)


class TestPivot:
def test_pivot(self):
Expand Down