Skip to content

Latest commit

 

History

History
1114 lines (927 loc) · 85.7 KB

v2.0.0.rst

File metadata and controls

1114 lines (927 loc) · 85.7 KB
Raw

What's new in 2.0.0 (??)

These are the changes in pandas 2.0.0. See release for a full changelog including other versions of pandas.

{{ header }}

Enhancements

Installing optional dependencies with pip extras

When installing pandas using pip, sets of optional dependencies can also be installed by specifying extras.

pip install "pandas[performance, aws]>=2.0.0"

The available extras, found in the installation guide<install.dependencies>, are [all, performance, computation, timezone, fss, aws, gcp, excel, parquet, feather, hdf5, spss, postgresql, mysql, sql-other, html, xml, plot, output_formatting, clipboard, compression, test] (39164).

Configuration option, mode.dtype_backend, to return pyarrow-backed dtypes

The use_nullable_dtypes keyword argument has been expanded to the following functions to enable automatic conversion to nullable dtypes (36712)

  • read_csv
  • read_clipboard
  • read_fwf
  • read_excel
  • read_html
  • read_xml
  • read_json
  • read_sql
  • read_sql_query
  • read_sql_table
  • read_orc
  • read_feather
  • to_numeric

Additionally a new global configuration, mode.dtype_backend can now be used in conjunction with the parameter use_nullable_dtypes=True in the following functions to select the nullable dtypes implementation.

  • read_csv (with engine="pyarrow" or engine="python")
  • read_clipboard (with engine="python")
  • read_excel
  • read_html
  • read_xml
  • read_json
  • read_parquet
  • read_orc
  • read_feather

And the following methods will also utilize the mode.dtype_backend option.

  • DataFrame.convert_dtypes
  • Series.convert_dtypes

By default, mode.dtype_backend is set to "pandas" to return existing, numpy-backed nullable dtypes, but it can also be set to "pyarrow" to return pyarrow-backed, nullable ArrowDtype (48957, 49997).

python

import io data = io.StringIO("""a,b,c,d,e,f,g,h,i 1,2.5,True,a,,,,, 3,4.5,False,b,6,7.5,True,a, """) with pd.option_context("mode.dtype_backend", "pandas"): df = pd.read_csv(data, use_nullable_dtypes=True) df.dtypes

data.seek(0) with pd.option_context("mode.dtype_backend", "pyarrow"): df_pyarrow = pd.read_csv(data, use_nullable_dtypes=True, engine="pyarrow") df_pyarrow.dtypes

Copy-on-Write improvements

  • A new lazy copy mechanism that defers the copy until the object in question is modified was added to the following methods:

    • DataFrame.reset_index / Series.reset_index
    • DataFrame.set_index
    • DataFrame.set_axis / Series.set_axis
    • DataFrame.rename_axis / Series.rename_axis
    • DataFrame.reindex / Series.reindex
    • DataFrame.reindex_like / Series.reindex_like
    • DataFrame.assign
    • DataFrame.drop
    • DataFrame.dropna / Series.dropna
    • DataFrame.select_dtypes
    • DataFrame.align / Series.align
    • Series.to_frame
    • DataFrame.rename / Series.rename
    • DataFrame.add_prefix / Series.add_prefix
    • DataFrame.add_suffix / Series.add_suffix
    • DataFrame.drop_duplicates / Series.drop_duplicates
    • DataFrame.reorder_levels / Series.reorder_levels

    These methods return views when Copy-on-Write is enabled, which provides a significant performance improvement compared to the regular execution (49473).

  • Accessing a single column of a DataFrame as a Series (e.g. df["col"]) now always returns a new object every time it is constructed when Copy-on-Write is enabled (not returning multiple times an identical, cached Series object). This ensures that those Series objects correctly follow the Copy-on-Write rules (49450)
  • The Series constructor will now create a lazy copy (deferring the copy until a modification to the data happens) when constructing a Series from an existing Series with the default of copy=False (50471)

Copy-on-Write can be enabled through

pd.set_option("mode.copy_on_write", True)
pd.options.mode.copy_on_write = True

Alternatively, copy on write can be enabled locally through:

with pd.option_context("mode.copy_on_write", True):
    ...

Other enhancements

  • read_sas now supports using encoding='infer' to correctly read and use the encoding specified by the sas file. (48048)
  • .DataFrameGroupBy.quantile, .SeriesGroupBy.quantile and .DataFrameGroupBy.std now preserve nullable dtypes instead of casting to numpy dtypes (37493)
  • Series.add_suffix, DataFrame.add_suffix, Series.add_prefix and DataFrame.add_prefix support an axis argument. If axis is set, the default behaviour of which axis to consider can be overwritten (47819)
  • assert_frame_equal now shows the first element where the DataFrames differ, analogously to pytest's output (47910)
  • Added index parameter to DataFrame.to_dict (46398)
  • Added support for extension array dtypes in merge (44240)
  • Added metadata propagation for binary operators on DataFrame (28283)
  • Added cumsum, cumprod, cummin and cummax to the ExtensionArray interface via _accumulate (28385)
  • .CategoricalConversionWarning, .InvalidComparison, .InvalidVersion, .LossySetitemError, and .NoBufferPresent are now exposed in pandas.errors (27656)
  • Fix test optional_extra by adding missing test package pytest-asyncio (48361)
  • DataFrame.astype exception message thrown improved to include column name when type conversion is not possible. (47571)
  • date_range now supports a unit keyword ("s", "ms", "us", or "ns") to specify the desired resolution of the output index (49106)
  • timedelta_range now supports a unit keyword ("s", "ms", "us", or "ns") to specify the desired resolution of the output index (49824)
  • DataFrame.to_json now supports a mode keyword with supported inputs 'w' and 'a'. Defaulting to 'w', 'a' can be used when lines=True and orient='records' to append record oriented json lines to an existing json file. (35849)
  • Added name parameter to IntervalIndex.from_breaks, IntervalIndex.from_arrays and IntervalIndex.from_tuples (48911)
  • Improve exception message when using assert_frame_equal on a DataFrame to include the column that is compared (50323)
  • Improved error message for merge_asof when join-columns were duplicated (50102)
  • Added Index.infer_objects analogous to Series.infer_objects (50034)
  • Added copy parameter to Series.infer_objects and DataFrame.infer_objects, passing False will avoid making copies for series or columns that are already non-object or where no better dtype can be inferred (50096)
  • DataFrame.plot.hist now recognizes xlabel and ylabel arguments (49793)
  • Improved error message in to_datetime for non-ISO8601 formats, informing users about the position of the first error (50361)
  • Improved error message when trying to align DataFrame objects (for example, in DataFrame.compare) to clarify that "identically labelled" refers to both index and columns (50083)
  • Added DatetimeIndex.as_unit and TimedeltaIndex.as_unit to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (50616)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

.DataFrameGroupBy.cumsum and .DataFrameGroupBy.cumprod overflow instead of lossy casting to float

In previous versions we cast to float when applying cumsum and cumprod which lead to incorrect results even if the result could be hold by int64 dtype. Additionally, the aggregation overflows consistent with numpy and the regular DataFrame.cumprod and DataFrame.cumsum methods when the limit of int64 is reached (37493).

Old Behavior

In [1]: df = pd.DataFrame({"key": ["b"] * 7, "value": 625})
In [2]: df.groupby("key")["value"].cumprod()[5]
Out[2]: 5.960464477539062e+16

We return incorrect results with the 6th value.

New Behavior

python

df = pd.DataFrame({"key": ["b"] * 7, "value": 625}) df.groupby("key")["value"].cumprod()

We overflow with the 7th value, but the 6th value is still correct.

.DataFrameGroupBy.nth and .SeriesGroupBy.nth now behave as filtrations

In previous versions of pandas, .DataFrameGroupBy.nth and .SeriesGroupBy.nth acted as if they were aggregations. However, for most inputs n, they may return either zero or multiple rows per group. This means that they are filtrations, similar to e.g. .DataFrameGroupBy.head. pandas now treats them as filtrations (13666).

python

df = pd.DataFrame({"a": [1, 1, 2, 1, 2], "b": [np.nan, 2.0, 3.0, 4.0, 5.0]}) gb = df.groupby("a")

Old Behavior

In [5]: gb.nth(n=1)
Out[5]:
   A    B
1  1  2.0
4  2  5.0

New Behavior

python

gb.nth(n=1)

In particular, the index of the result is derived from the input by selecting the appropriate rows. Also, when n is larger than the group, no rows instead of NaN is returned.

Old Behavior

In [5]: gb.nth(n=3, dropna="any")
Out[5]:
    B
A
1 NaN
2 NaN

New Behavior

python

gb.nth(n=3, dropna="any")

Backwards incompatible API changes

Construction with datetime64 or timedelta64 dtype with unsupported resolution

In past versions, when constructing a Series or DataFrame and passing a "datetime64" or "timedelta64" dtype with unsupported resolution (i.e. anything other than "ns"), pandas would silently replace the given dtype with its nanosecond analogue:

Previous behavior:

In [5]: pd.Series(["2016-01-01"], dtype="datetime64[s]")
Out[5]:
0   2016-01-01
dtype: datetime64[ns]

In [6] pd.Series(["2016-01-01"], dtype="datetime64[D]")
Out[6]:
0   2016-01-01
dtype: datetime64[ns]

In pandas 2.0 we support resolutions "s", "ms", "us", and "ns". When passing a supported dtype (e.g. "datetime64[s]"), the result now has exactly the requested dtype:

New behavior:

python

pd.Series(["2016-01-01"], dtype="datetime64[s]")

With an un-supported dtype, pandas now raises instead of silently swapping in a supported dtype:

New behavior:

python

pd.Series(["2016-01-01"], dtype="datetime64[D]")

Disallow astype conversion to non-supported datetime64/timedelta64 dtypes

In previous versions, converting a Series or DataFrame from datetime64[ns] to a different datetime64[X] dtype would return with datetime64[ns] dtype instead of the requested dtype. In pandas 2.0, support is added for "datetime64[s]", "datetime64[ms]", and "datetime64[us]" dtypes, so converting to those dtypes gives exactly the requested dtype:

Previous behavior:

python

idx = pd.date_range("2016-01-01", periods=3) ser = pd.Series(idx)

Previous behavior:

In [4]: ser.astype("datetime64[s]")
Out[4]:
0   2016-01-01
1   2016-01-02
2   2016-01-03
dtype: datetime64[ns]

With the new behavior, we get exactly the requested dtype:

New behavior:

python

ser.astype("datetime64[s]")

For non-supported resolutions e.g. "datetime64[D]", we raise instead of silently ignoring the requested dtype:

New behavior:

python

ser.astype("datetime64[D]")

For conversion from timedelta64[ns] dtypes, the old behavior converted to a floating point format.

Previous behavior:

python

idx = pd.timedelta_range("1 Day", periods=3) ser = pd.Series(idx)

Previous behavior:

In [7]: ser.astype("timedelta64[s]")
Out[7]:
0     86400.0
1    172800.0
2    259200.0
dtype: float64

In [8]: ser.astype("timedelta64[D]")
Out[8]:
0    1.0
1    2.0
2    3.0
dtype: float64

The new behavior, as for datetime64, either gives exactly the requested dtype or raises:

New behavior:

python

ser.astype("timedelta64[s]") ser.astype("timedelta64[D]")

UTC and fixed-offset timezones default to standard-library tzinfo objects

In previous versions, the default tzinfo object used to represent UTC was pytz.UTC. In pandas 2.0, we default to datetime.timezone.utc instead. Similarly, for timezones represent fixed UTC offsets, we use datetime.timezone objects instead of pytz.FixedOffset objects. See (34916)

Previous behavior:

In [2]: ts = pd.Timestamp("2016-01-01", tz="UTC")
In [3]: type(ts.tzinfo)
Out[3]: pytz.UTC

In [4]: ts2 = pd.Timestamp("2016-01-01 04:05:06-07:00")
In [3]: type(ts2.tzinfo)
Out[5]: pytz._FixedOffset

New behavior:

python

ts = pd.Timestamp("2016-01-01", tz="UTC") type(ts.tzinfo)

ts2 = pd.Timestamp("2016-01-01 04:05:06-07:00") type(ts2.tzinfo)

For timezones that are neither UTC nor fixed offsets, e.g. "US/Pacific", we continue to default to pytz objects.

Empty DataFrames/Series will now default to have a RangeIndex

Before, constructing an empty (where data is None or an empty list-like argument) Series or DataFrame without specifying the axes (index=None, columns=None) would return the axes as empty Index with object dtype.

Now, the axes return an empty RangeIndex.

Previous behavior:

In [8]: pd.Series().index
Out[8]:
Index([], dtype='object')

In [9] pd.DataFrame().axes
Out[9]:
[Index([], dtype='object'), Index([], dtype='object')]

New behavior:

python

pd.Series().index pd.DataFrame().axes

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package Minimum Version Required Changed
mypy (dev) 0.991

X
pytest (dev) 7.0.0

X
pytest-xdist (dev) 2.2.0

X
hypothesis (dev) 6.34.2

X
python-dateutil 2.8.2

X

X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package Minimum Version Changed
pyarrow 6.0.0

X
matplotlib 3.6.1

X
fastparquet 0.6.3

X
xarray 0.21.0

X

See install.dependencies and install.optional_dependencies for more.

Datetimes are now parsed with a consistent format

In the past, to_datetime guessed the format for each element independently. This was appropriate for some cases where elements had mixed date formats - however, it would regularly cause problems when users expected a consistent format but the function would switch formats between elements. As of version 2.0.0, parsing will use a consistent format, determined by the first non-NA value (unless the user specifies a format, in which case that is used).

Old behavior:

In [1]: ser = pd.Series(['13-01-2000', '12-01-2000'])
In [2]: pd.to_datetime(ser)
Out[2]:
0   2000-01-13
1   2000-12-01
dtype: datetime64[ns]

New behavior:

python

ser = pd.Series(['13-01-2000', '12-01-2000']) pd.to_datetime(ser)

Note that this affects read_csv as well.

If you still need to parse dates with inconsistent formats, you'll need to apply to_datetime to each element individually, e.g. :

ser = pd.Series(['13-01-2000', '12 January 2000'])
ser.apply(pd.to_datetime)

Other API changes

  • The freq, tz, nanosecond, and unit keywords in the Timestamp constructor are now keyword-only (45307, 32526)
  • Passing nanoseconds greater than 999 or less than 0 in Timestamp now raises a ValueError (48538, 48255)
  • read_csv: specifying an incorrect number of columns with index_col of now raises ParserError instead of IndexError when using the c parser.
  • Default value of dtype in get_dummies is changed to bool from uint8 (45848)
  • DataFrame.astype, Series.astype, and DatetimeIndex.astype casting datetime64 data to any of "datetime64[s]", "datetime64[ms]", "datetime64[us]" will return an object with the given resolution instead of coercing back to "datetime64[ns]" (48928)
  • DataFrame.astype, Series.astype, and DatetimeIndex.astype casting timedelta64 data to any of "timedelta64[s]", "timedelta64[ms]", "timedelta64[us]" will return an object with the given resolution instead of coercing to "float64" dtype (48963)
  • DatetimeIndex.astype, TimedeltaIndex.astype, PeriodIndex.astype Series.astype, DataFrame.astype with datetime64, timedelta64 or PeriodDtype dtypes no longer allow converting to integer dtypes other than "int64", do obj.astype('int64', copy=False).astype(dtype) instead (49715)
  • Index.astype now allows casting from float64 dtype to datetime-like dtypes, matching Series behavior (49660)
  • Passing data with dtype of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to TimedeltaIndex, Series, or DataFrame constructors will now retain that dtype instead of casting to "timedelta64[ns]"; timedelta64 data with lower resolution will be cast to the lowest supported resolution "timedelta64[s]" (49014)
  • Passing dtype of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to TimedeltaIndex, Series, or DataFrame constructors will now retain that dtype instead of casting to "timedelta64[ns]"; passing a dtype with lower resolution for Series or DataFrame will be cast to the lowest supported resolution "timedelta64[s]" (49014)
  • Passing a np.datetime64 object with non-nanosecond resolution to Timestamp will retain the input resolution if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (49008)
  • Passing datetime64 values with resolution other than nanosecond to to_datetime will retain the input resolution if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (50369)
  • Passing a string in ISO-8601 format to Timestamp will retain the resolution of the parsed input if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (49737)
  • The other argument in DataFrame.mask and Series.mask now defaults to no_default instead of np.nan consistent with DataFrame.where and Series.where. Entries will be filled with the corresponding NULL value (np.nan for numpy dtypes, pd.NA for extension dtypes). (49111)
  • Changed behavior of Series.quantile and DataFrame.quantile with SparseDtype to retain sparse dtype (49583)
  • When creating a Series with a object-dtype Index of datetime objects, pandas no longer silently converts the index to a DatetimeIndex (39307, 23598)
  • Series.unique with dtype "timedelta64[ns]" or "datetime64[ns]" now returns TimedeltaArray or DatetimeArray instead of numpy.ndarray (49176)
  • to_datetime and DatetimeIndex now allow sequences containing both datetime objects and numeric entries, matching Series behavior (49037, 50453)
  • pandas.api.dtypes.is_string_dtype now only returns True for array-likes with dtype=object when the elements are inferred to be strings (15585)
  • Passing a sequence containing datetime objects and date objects to Series constructor will return with object dtype instead of datetime64[ns] dtype, consistent with Index behavior (49341)
  • Passing strings that cannot be parsed as datetimes to Series or DataFrame with dtype="datetime64[ns]" will raise instead of silently ignoring the keyword and returning object dtype (24435)
  • Passing a sequence containing a type that cannot be converted to Timedelta to to_timedelta or to the Series or DataFrame constructor with dtype="timedelta64[ns]" or to TimedeltaIndex now raises TypeError instead of ValueError (49525)
  • Changed behavior of Index constructor with sequence containing at least one NaT and everything else either None or NaN to infer datetime64[ns] dtype instead of object, matching Series behavior (49340)
  • read_stata with parameter index_col set to None (the default) will now set the index on the returned DataFrame to a RangeIndex instead of a Int64Index (49745)
  • Changed behavior of Index, Series, and DataFrame arithmetic methods when working with object-dtypes, the results no longer do type inference on the result of the array operations, use result.infer_objects(copy=False) to do type inference on the result (49999, 49714)
  • Changed behavior of Index constructor with an object-dtype numpy.ndarray containing all-bool values or all-complex values, this will now retain object dtype, consistent with the Series behavior (49594)
  • Added "None" to default na_values in read_csv (50286)
  • Changed behavior of Series and DataFrame constructors when given an integer dtype and floating-point data that is not round numbers, this now raises ValueError instead of silently retaining the float dtype; do Series(data) or DataFrame(data) to get the old behavior, and Series(data).astype(dtype) or DataFrame(data).astype(dtype) to get the specified dtype (49599)
  • Changed behavior of DataFrame.shift with axis=1, an integer fill_value, and homogeneous datetime-like dtype, this now fills new columns with integer dtypes instead of casting to datetimelike (49842)
  • Files are now closed when encountering an exception in read_json (49921)
  • Changed behavior of read_csv, read_json & read_fwf, where the index will now always be a RangeIndex, when no index is specified. Previously the index would be a Index with dtype object if the new DataFrame/Series has length 0 (49572)
  • DataFrame.values, DataFrame.to_numpy, DataFrame.xs, DataFrame.reindex, DataFrame.fillna, and DataFrame.replace no longer silently consolidate the underlying arrays; do df = df.copy() to ensure consolidation (49356)
  • Creating a new DataFrame using a full slice on both axes with ~DataFrame.loc or ~DataFrame.iloc (thus, df.loc[:, :] or df.iloc[:, :]) now returns a new DataFrame (shallow copy) instead of the original DataFrame, consistent with other methods to get a full slice (for example df.loc[:] or df[:]) (49469)
  • Disallow computing cumprod for Timedelta object; previously this returned incorrect values (50246)
  • Loading a JSON file with duplicate columns using read_json(orient='split') renames columns to avoid duplicates, as read_csv and the other readers do (50370)
  • to_datetime with unit of either "Y" or "M" will now raise if a sequence contains a non-round float value, matching the Timestamp behavior (50301)

Deprecations

  • Deprecated argument infer_datetime_format in to_datetime and read_csv, as a strict version of it is now the default (48621)
  • Deprecated pandas.io.sql.execute (50185)
  • Index.is_boolean has been deprecated. Use pandas.api.types.is_bool_dtype instead (50042)
  • Index.is_integer has been deprecated. Use pandas.api.types.is_integer_dtype instead (50042)
  • Index.is_floating has been deprecated. Use pandas.api.types.is_float_dtype instead (50042)
  • Index.holds_integer has been deprecated. Use pandas.api.types.infer_dtype instead (50243)
  • Index.is_categorical has been deprecated. Use pandas.api.types.is_categorical_dtype instead (50042)

Removal of prior version deprecations/changes

  • Removed deprecated Timestamp.freq, Timestamp.freqstr and argument freq from the Timestamp constructor and Timestamp.fromordinal (14146)
  • Removed deprecated CategoricalBlock, Block.is_categorical, require datetime64 and timedelta64 values to be wrapped in DatetimeArray or TimedeltaArray before passing to Block.make_block_same_class, require DatetimeTZBlock.values to have the correct ndim when passing to the BlockManager constructor, and removed the "fastpath" keyword from the SingleBlockManager constructor (40226, 40571)
  • Removed deprecated global option use_inf_as_null in favor of use_inf_as_na (17126)
  • Removed deprecated module pandas.core.index (30193)
  • Removed deprecated alias pandas.core.tools.datetimes.to_time, import the function directly from pandas.core.tools.times instead (34145)
  • Removed deprecated alias pandas.io.json.json_normalize, import the function directly from pandas.json_normalize instead (27615)
  • Removed deprecated Categorical.to_dense, use np.asarray(cat) instead (32639)
  • Removed deprecated Categorical.take_nd (27745)
  • Removed deprecated Categorical.mode, use Series(cat).mode() instead (45033)
  • Removed deprecated Categorical.is_dtype_equal and CategoricalIndex.is_dtype_equal (37545)
  • Removed deprecated CategoricalIndex.take_nd (30702)
  • Removed deprecated Index.is_type_compatible (42113)
  • Removed deprecated Index.is_mixed, check index.inferred_type directly instead (32922)
  • Removed deprecated pandas.api.types.is_categorical; use pandas.api.types.is_categorical_dtype instead (33385)
  • Removed deprecated Index.asi8 (37877)
  • Enforced deprecation changing behavior when passing datetime64[ns] dtype data and timezone-aware dtype to Series, interpreting the values as wall-times instead of UTC times, matching DatetimeIndex behavior (41662)
  • Enforced deprecation changing behavior when applying a numpy ufunc on multiple non-aligned (on the index or columns) DataFrame that will now align the inputs first (39239)
  • Removed deprecated DataFrame._AXIS_NUMBERS, DataFrame._AXIS_NAMES, Series._AXIS_NUMBERS, Series._AXIS_NAMES (33637)
  • Removed deprecated Index.to_native_types, use obj.astype(str) instead (36418)
  • Removed deprecated Series.iteritems, DataFrame.iteritems, use obj.items instead (45321)
  • Removed deprecated DataFrame.lookup (35224)
  • Removed deprecated Series.append, DataFrame.append, use concat instead (35407)
  • Removed deprecated Series.iteritems, DataFrame.iteritems and HDFStore.iteritems use obj.items instead (45321)
  • Removed deprecated DatetimeIndex.union_many (45018)
  • Removed deprecated weekofyear and week attributes of DatetimeArray, DatetimeIndex and dt accessor in favor of isocalendar().week (33595)
  • Removed deprecated RangeIndex._start, RangeIndex._stop, RangeIndex._step, use start, stop, step instead (30482)
  • Removed deprecated DatetimeIndex.to_perioddelta, Use dtindex - dtindex.to_period(freq).to_timestamp() instead (34853)
  • Removed deprecated .Styler.hide_index and .Styler.hide_columns (49397)
  • Removed deprecated .Styler.set_na_rep and .Styler.set_precision (49397)
  • Removed deprecated .Styler.where (49397)
  • Removed deprecated .Styler.render (49397)
  • Removed deprecated argument null_color in .Styler.highlight_null (49397)
  • Removed deprecated argument check_less_precise in .testing.assert_frame_equal, .testing.assert_extension_array_equal, .testing.assert_series_equal, .testing.assert_index_equal (30562)
  • Removed deprecated null_counts argument in DataFrame.info. Use show_counts instead (37999)
  • Removed deprecated Index.is_monotonic, and Series.is_monotonic; use obj.is_monotonic_increasing instead (45422)
  • Removed deprecated Index.is_all_dates (36697)
  • Enforced deprecation disallowing passing a timezone-aware Timestamp and dtype="datetime64[ns]" to Series or DataFrame constructors (41555)
  • Enforced deprecation disallowing passing a sequence of timezone-aware values and dtype="datetime64[ns]" to to Series or DataFrame constructors (41555)
  • Enforced deprecation disallowing numpy.ma.mrecords.MaskedRecords in the DataFrame constructor; pass "{name: data[name] for name in data.dtype.names} instead (40363)
  • Enforced deprecation disallowing unit-less "datetime64" dtype in Series.astype and DataFrame.astype (47844)
  • Enforced deprecation disallowing using .astype to convert a datetime64[ns] Series, DataFrame, or DatetimeIndex to timezone-aware dtype, use obj.tz_localize or ser.dt.tz_localize instead (39258)
  • Enforced deprecation disallowing using .astype to convert a timezone-aware Series, DataFrame, or DatetimeIndex to timezone-naive datetime64[ns] dtype, use obj.tz_localize(None) or obj.tz_convert("UTC").tz_localize(None) instead (39258)
  • Enforced deprecation disallowing passing non boolean argument to sort in concat (44629)
  • Removed Date parser functions ~pandas.io.date_converters.parse_date_time, ~pandas.io.date_converters.parse_date_fields, ~pandas.io.date_converters.parse_all_fields and ~pandas.io.date_converters.generic_parser (24518)
  • Removed argument index from the core.arrays.SparseArray constructor (43523)
  • Remove argument squeeze from DataFrame.groupby and Series.groupby (32380)
  • Removed deprecated apply, apply_index, __call__, onOffset, and isAnchored attributes from DateOffset (34171)
  • Removed keep_tz argument in DatetimeIndex.to_series (29731)
  • Remove arguments names and dtype from Index.copy and levels and codes from MultiIndex.copy (35853, 36685)
  • Remove argument inplace from MultiIndex.set_levels and MultiIndex.set_codes (35626)
  • Removed arguments verbose and encoding from DataFrame.to_excel and Series.to_excel (47912)
  • Removed argument line_terminator from DataFrame.to_csv and Series.to_csv, use lineterminator instead (45302)
  • Removed argument inplace from DataFrame.set_axis and Series.set_axis, use obj = obj.set_axis(..., copy=False) instead (48130)
  • Disallow passing positional arguments to MultiIndex.set_levels and MultiIndex.set_codes (41485)
  • Disallow parsing to Timedelta strings with components with units "Y", "y", or "M", as these do not represent unambiguous durations (36838)
  • Removed MultiIndex.is_lexsorted and MultiIndex.lexsort_depth (38701)
  • Removed argument how from PeriodIndex.astype, use PeriodIndex.to_timestamp instead (37982)
  • Removed argument try_cast from DataFrame.mask, DataFrame.where, Series.mask and Series.where (38836)
  • Removed argument tz from Period.to_timestamp, use obj.to_timestamp(...).tz_localize(tz) instead (34522)
  • Removed argument sort_columns in DataFrame.plot and Series.plot (47563)
  • Removed argument is_copy from DataFrame.take and Series.take (30615)
  • Removed argument kind from Index.get_slice_bound, Index.slice_indexer and Index.slice_locs (41378)
  • Removed arguments prefix, squeeze, error_bad_lines and warn_bad_lines from read_csv (40413, 43427)
  • Removed argument datetime_is_numeric from DataFrame.describe and Series.describe as datetime data will always be summarized as numeric data (34798)
  • Disallow passing list key to Series.xs and DataFrame.xs, pass a tuple instead (41789)
  • Disallow subclass-specific keywords (e.g. "freq", "tz", "names", "closed") in the Index constructor (38597)
  • Removed argument inplace from Categorical.remove_unused_categories (37918)
  • Disallow passing non-round floats to Timestamp with unit="M" or unit="Y" (47266)
  • Remove keywords convert_float and mangle_dupe_cols from read_excel (41176)
  • Remove keyword mangle_dupe_cols from read_csv and read_table (48137)
  • Removed errors keyword from DataFrame.where, Series.where, DataFrame.mask and Series.mask (47728)
  • Disallow passing non-keyword arguments to read_excel except io and sheet_name (34418)
  • Disallow passing non-keyword arguments to DataFrame.drop and Series.drop except labels (41486)
  • Disallow passing non-keyword arguments to DataFrame.fillna and Series.fillna except value (41485)
  • Disallow passing non-keyword arguments to StringMethods.split and StringMethods.rsplit except for pat (47448)
  • Disallow passing non-keyword arguments to DataFrame.set_index except keys (41495)
  • Disallow passing non-keyword arguments to Resampler.interpolate except method (41699)
  • Disallow passing non-keyword arguments to DataFrame.reset_index and Series.reset_index except level (41496)
  • Disallow passing non-keyword arguments to DataFrame.dropna and Series.dropna (41504)
  • Disallow passing non-keyword arguments to ExtensionArray.argsort (46134)
  • Disallow passing non-keyword arguments to Categorical.sort_values (47618)
  • Disallow passing non-keyword arguments to Index.drop_duplicates and Series.drop_duplicates (41485)
  • Disallow passing non-keyword arguments to DataFrame.drop_duplicates except for subset (41485)
  • Disallow passing non-keyword arguments to DataFrame.sort_index and Series.sort_index (41506)
  • Disallow passing non-keyword arguments to DataFrame.interpolate and Series.interpolate except for method (41510)
  • Disallow passing non-keyword arguments to DataFrame.any and Series.any (44896)
  • Disallow passing non-keyword arguments to Index.set_names except for names (41551)
  • Disallow passing non-keyword arguments to Index.join except for other (46518)
  • Disallow passing non-keyword arguments to concat except for objs (41485)
  • Disallow passing non-keyword arguments to pivot except for data (48301)
  • Disallow passing non-keyword arguments to DataFrame.pivot (48301)
  • Disallow passing non-keyword arguments to read_html except for io (27573)
  • Disallow passing non-keyword arguments to read_json except for path_or_buf (27573)
  • Disallow passing non-keyword arguments to read_sas except for filepath_or_buffer (47154)
  • Disallow passing non-keyword arguments to read_stata except for filepath_or_buffer (48128)
  • Disallow passing non-keyword arguments to read_csv except filepath_or_buffer (41485)
  • Disallow passing non-keyword arguments to read_table except filepath_or_buffer (41485)
  • Disallow passing non-keyword arguments to read_fwf except filepath_or_buffer (44710)
  • Disallow passing non-keyword arguments to read_xml except for path_or_buffer (45133)
  • Disallow passing non-keyword arguments to Series.mask and DataFrame.mask except cond and other (41580)
  • Disallow passing non-keyword arguments to DataFrame.to_stata except for path (48128)
  • Disallow passing non-keyword arguments to DataFrame.where and Series.where except for cond and other (41523)
  • Disallow passing non-keyword arguments to Series.set_axis and DataFrame.set_axis except for labels (41491)
  • Disallow passing non-keyword arguments to Series.rename_axis and DataFrame.rename_axis except for mapper (47587)
  • Disallow passing non-keyword arguments to Series.clip and DataFrame.clip (41511)
  • Disallow passing non-keyword arguments to Series.bfill, Series.ffill, DataFrame.bfill and DataFrame.ffill (41508)
  • Disallow passing non-keyword arguments to DataFrame.replace, Series.replace except for to_replace and value (47587)
  • Disallow passing non-keyword arguments to DataFrame.sort_values except for by (41505)
  • Disallow passing non-keyword arguments to Series.sort_values (41505)
  • Disallow passing 2 non-keyword arguments to DataFrame.reindex (17966)
  • Disallow Index.reindex with non-unique Index objects (42568)
  • Disallowed constructing Categorical with scalar data (38433)
  • Disallowed constructing CategoricalIndex without passing data (38944)
  • Removed .Rolling.validate, .Expanding.validate, and .ExponentialMovingWindow.validate (43665)
  • Removed Rolling.win_type returning "freq" (38963)
  • Removed Rolling.is_datetimelike (38963)
  • Removed the level keyword in DataFrame and Series aggregations; use groupby instead (39983)
  • Removed deprecated Timedelta.delta, Timedelta.is_populated, and Timedelta.freq (46430, 46476)
  • Removed deprecated NaT.freq (45071)
  • Removed deprecated Categorical.replace, use Series.replace instead (44929)
  • Removed the numeric_only keyword from Categorical.min and Categorical.max in favor of skipna (48821)
  • Changed behavior of DataFrame.median and DataFrame.mean with numeric_only=None to not exclude datetime-like columns THIS NOTE WILL BE IRRELEVANT ONCE numeric_only=None DEPRECATION IS ENFORCED (29941)
  • Removed is_extension_type in favor of is_extension_array_dtype (29457)
  • Removed .ExponentialMovingWindow.vol (39220)
  • Removed Index.get_value and Index.set_value (33907, 28621)
  • Removed Series.slice_shift and DataFrame.slice_shift (37601)
  • Remove DataFrameGroupBy.pad and DataFrameGroupBy.backfill (45076)
  • Remove numpy argument from read_json (30636)
  • Disallow passing abbreviations for orient in DataFrame.to_dict (32516)
  • Disallow partial slicing on an non-monotonic DatetimeIndex with keys which are not in Index. This now raises a KeyError (18531)
  • Removed get_offset in favor of to_offset (30340)
  • Removed the warn keyword in infer_freq (45947)
  • Removed the include_start and include_end arguments in DataFrame.between_time in favor of inclusive (43248)
  • Removed the closed argument in date_range and bdate_range in favor of inclusive argument (40245)
  • Removed the center keyword in DataFrame.expanding (20647)
  • Removed the truediv keyword from eval (29812)
  • Removed the method and tolerance arguments in Index.get_loc. Use index.get_indexer([label], method=..., tolerance=...) instead (42269)
  • Removed the pandas.datetime submodule (30489)
  • Removed the pandas.np submodule (30296)
  • Removed pandas.util.testing in favor of pandas.testing (30745)
  • Removed Series.str.__iter__ (28277)
  • Removed pandas.SparseArray in favor of arrays.SparseArray (30642)
  • Removed pandas.SparseSeries and pandas.SparseDataFrame, including pickle support. (30642)
  • Enforced disallowing passing an integer fill_value to DataFrame.shift and Series.shift with datetime64, timedelta64, or period dtypes (:issue:`32591)
  • Enforced disallowing a string column label into times in DataFrame.ewm (43265)
  • Enforced disallowing passing True and False into inclusive in Series.between in favor of "both" and "neither" respectively (40628)
  • Enforced disallowing using usecols with out of bounds indices for read_csv with engine="c" (25623)
  • Enforced disallowing the use of **kwargs in .ExcelWriter; use the keyword argument engine_kwargs instead (40430)
  • Enforced disallowing a tuple of column labels into .DataFrameGroupBy.__getitem__ (30546)
  • Enforced disallowing missing labels when indexing with a sequence of labels on a level of a MultiIndex. This now raises a KeyError (42351)
  • Enforced disallowing setting values with .loc using a positional slice. Use .loc with labels or .iloc with positions instead (31840)
  • Enforced disallowing positional indexing with a float key even if that key is a round number, manually cast to integer instead (34193)
  • Enforced disallowing using a DataFrame indexer with .iloc, use .loc instead for automatic alignment (39022)
  • Enforced disallowing set or dict indexers in __getitem__ and __setitem__ methods (42825)
  • Enforced disallowing indexing on a Index or positional indexing on a Series producing multi-dimensional objects e.g. obj[:, None], convert to numpy before indexing instead (35141)
  • Enforced disallowing dict or set objects in suffixes in merge (34810)
  • Enforced disallowing merge to produce duplicated columns through the suffixes keyword and already existing columns (22818)
  • Enforced disallowing using merge or join on a different number of levels (34862)
  • Enforced disallowing value_name argument in DataFrame.melt to match an element in the DataFrame columns (35003)
  • Enforced disallowing passing showindex into **kwargs in DataFrame.to_markdown and Series.to_markdown in favor of index (33091)
  • Removed setting Categorical._codes directly (41429)
  • Removed setting Categorical.categories directly (47834)
  • Removed argument inplace from Categorical.add_categories, Categorical.remove_categories, Categorical.set_categories, Categorical.rename_categories, Categorical.reorder_categories, Categorical.set_ordered, Categorical.as_ordered, Categorical.as_unordered (37981, 41118, 41133, 47834)
  • Enforced Rolling.count with min_periods=None to default to the size of the window (31302)
  • Renamed fname to path in DataFrame.to_parquet, DataFrame.to_stata and DataFrame.to_feather (30338)
  • Enforced disallowing indexing a Series with a single item list with a slice (e.g. ser[[slice(0, 2)]]). Either convert the list to tuple, or pass the slice directly instead (31333)
  • Changed behavior indexing on a DataFrame with a DatetimeIndex index using a string indexer, previously this operated as a slice on rows, now it operates like any other column key; use frame.loc[key] for the old behavior (36179)
  • Enforced the display.max_colwidth option to not accept negative integers (31569)
  • Removed the display.column_space option in favor of df.to_string(col_space=...) (47280)
  • Removed the deprecated method mad from pandas classes (11787)
  • Removed the deprecated method tshift from pandas classes (11631)
  • Changed behavior of empty data passed into Series; the default dtype will be object instead of float64 (29405)
  • Changed the behavior of DatetimeIndex.union, DatetimeIndex.intersection, and DatetimeIndex.symmetric_difference with mismatched timezones to convert to UTC instead of casting to object dtype (39328)
  • Changed the behavior of to_datetime with argument "now" with utc=False to match Timestamp("now") (18705)
  • Changed the behavior of indexing on a timezone-aware DatetimeIndex with a timezone-naive datetime object or vice-versa; these now behave like any other non-comparable type by raising KeyError (36148)
  • Changed the behavior of Index.reindex, Series.reindex, and DataFrame.reindex with a datetime64 dtype and a datetime.date object for fill_value; these are no longer considered equivalent to datetime.datetime objects so the reindex casts to object dtype (39767)
  • Changed behavior of SparseArray.astype when given a dtype that is not explicitly SparseDtype, cast to the exact requested dtype rather than silently using a SparseDtype instead (34457)
  • Changed behavior of Index.ravel to return a view on the original Index instead of a np.ndarray (36900)
  • Changed behavior of Series.to_frame and Index.to_frame with explicit name=None to use None for the column name instead of the index's name or default 0 (45523)
  • Changed behavior of concat with one array of bool-dtype and another of integer dtype, this now returns object dtype instead of integer dtype; explicitly cast the bool object to integer before concatenating to get the old behavior (45101)
  • Changed behavior of DataFrame constructor given floating-point data and an integer dtype, when the data cannot be cast losslessly, the floating point dtype is retained, matching Series behavior (41170)
  • Changed behavior of Index constructor when given a np.ndarray with object-dtype containing numeric entries; this now retains object dtype rather than inferring a numeric dtype, consistent with Series behavior (42870)
  • Changed behavior of Index.__and__, Index.__or__ and Index.__xor__ to behave as logical operations (matching Series behavior) instead of aliases for set operations (37374)
  • Changed behavior of DataFrame constructor when passed a list whose first element is a Categorical, this now treats the elements as rows casting to object dtype, consistent with behavior for other types (38845)
  • Changed behavior of DataFrame constructor when passed a dtype (other than int) that the data cannot be cast to; it now raises instead of silently ignoring the dtype (41733)
  • Changed the behavior of Series constructor, it will no longer infer a datetime64 or timedelta64 dtype from string entries (41731)
  • Changed behavior of Timestamp constructor with a np.datetime64 object and a tz passed to interpret the input as a wall-time as opposed to a UTC time (42288)
  • Changed behavior of Timestamp.utcfromtimestamp to return a timezone-aware object satisfying Timestamp.utcfromtimestamp(val).timestamp() == val (45083)
  • Changed behavior of Index constructor when passed a SparseArray or SparseDtype to retain that dtype instead of casting to numpy.ndarray (43930)
  • Changed behavior of setitem-like operations (__setitem__, fillna, where, mask, replace, insert, fill_value for shift) on an object with DatetimeTZDtype when using a value with a non-matching timezone, the value will be cast to the object's timezone instead of casting both to object-dtype (44243)
  • Changed behavior of Index, Series, DataFrame constructors with floating-dtype data and a DatetimeTZDtype, the data are now interpreted as UTC-times instead of wall-times, consistent with how integer-dtype data are treated (45573)
  • Changed behavior of Series and DataFrame constructors with integer dtype and floating-point data containing NaN, this now raises IntCastingNaNError (40110)
  • Changed behavior of Series and DataFrame constructors with an integer dtype and values that are too large to losslessly cast to this dtype, this now raises ValueError (41734)
  • Changed behavior of Series and DataFrame constructors with an integer dtype and values having either datetime64 or timedelta64 dtypes, this now raises TypeError, use values.view("int64") instead (41770)
  • Removed the deprecated base and loffset arguments from pandas.DataFrame.resample, pandas.Series.resample and pandas.Grouper. Use offset or origin instead (31809)
  • Changed behavior of Series.fillna and DataFrame.fillna with timedelta64[ns] dtype and an incompatible fill_value; this now casts to object dtype instead of raising, consistent with the behavior with other dtypes (45746)
  • Change the default argument of regex for Series.str.replace from True to False. Additionally, a single character pat with regex=True is now treated as a regular expression instead of a string literal. (36695, 24804)
  • Changed behavior of DataFrame.any and DataFrame.all with bool_only=True; object-dtype columns with all-bool values will no longer be included, manually cast to bool dtype first (46188)
  • Changed behavior of DataFrame.max, DataFrame.min, DataFrame.mean, DataFrame.median, DataFrame.skew, DataFrame.kurt with axis=None to return a scalar applying the aggregation across both axes (45072)
  • Changed behavior of comparison of a Timestamp with a datetime.date object; these now compare as un-equal and raise on inequality comparisons, matching the datetime.datetime behavior (36131)
  • Changed behavior of comparison of NaT with a datetime.date object; these now raise on inequality comparisons (39196)
  • Enforced deprecation of silently dropping columns that raised a TypeError in Series.transform and DataFrame.transform when used with a list or dictionary (43740)
  • Changed behavior of DataFrame.apply with list-like so that any partial failure will raise an error (43740)
  • Changed behavior of Series.__setitem__ with an integer key and a Float64Index when the key is not present in the index; previously we treated the key as positional (behaving like series.iloc[key] = val), now we treat it is a label (behaving like series.loc[key] = val), consistent with Series.__getitem__ behavior (:issue:`33469)
  • Removed na_sentinel argument from factorize, .Index.factorize, and .ExtensionArray.factorize (47157)
  • Changed behavior of Series.diff and DataFrame.diff with ExtensionDtype dtypes whose arrays do not implement diff, these now raise TypeError rather than casting to numpy (31025)
  • Enforced deprecation of calling numpy "ufunc"s on DataFrame with method="outer"; this now raises NotImplementedError (36955)
  • Enforced deprecation disallowing passing numeric_only=True to Series reductions (rank, any, all, ...) with non-numeric dtype (47500)
  • Changed behavior of DataFrameGroupBy.apply and SeriesGroupBy.apply so that group_keys is respected even if a transformer is detected (34998)
  • Comparisons between a DataFrame and a Series where the frame's columns do not match the series's index raise ValueError instead of automatically aligning, do left, right = left.align(right, axis=1, copy=False) before comparing (36795)
  • Enforced deprecation numeric_only=None (the default) in DataFrame reductions that would silently drop columns that raised; numeric_only now defaults to False (41480)
  • Changed default of numeric_only to False in all DataFrame methods with that argument (46096, 46906)
  • Changed default of numeric_only to False in Series.rank (47561)
  • Enforced deprecation of silently dropping nuisance columns in groupby and resample operations when numeric_only=False (41475)
  • Enforced deprecation of silently dropping nuisance columns in Rolling, Expanding, and ExponentialMovingWindow ops. This will now raise a .errors.DataError (42834)
  • Changed behavior in setting values with df.loc[:, foo] = bar or df.iloc[:, foo] = bar, these now always attempt to set values inplace before falling back to casting (45333)
  • Changed default of numeric_only in various .DataFrameGroupBy methods; all methods now default to numeric_only=False (46072)
  • Changed default of numeric_only to False in .Resampler methods (47177)
  • Using the method DataFrameGroupBy.transform with a callable that returns DataFrames will align to the input's index (47244)
  • When providing a list of columns of length one to DataFrame.groupby, the keys that are returned by iterating over the resulting DataFrameGroupBy object will now be tuples of length one (47761)
  • Removed deprecated methods ExcelWriter.write_cells, ExcelWriter.save, ExcelWriter.cur_sheet, ExcelWriter.handles, ExcelWriter.path (45795)
  • The ExcelWriter attribute book can no longer be set; it is still available to be accessed and mutated (48943)
  • Removed unused *args and **kwargs in Rolling, Expanding, and ExponentialMovingWindow ops (47851)
  • Removed the deprecated argument line_terminator from DataFrame.to_csv (45302)
  • Removed the deprecated argument label from lreshape (30219)
  • Arguments after expr in DataFrame.eval and DataFrame.query are keyword-only (47587)
  • Removed Index._get_attributes_dict (50648)
  • Removed Series.__array_wrap__ (50648)

Performance improvements

  • Performance improvement in .DataFrameGroupBy.median and .SeriesGroupBy.median and .DataFrameGroupBy.cumprod for nullable dtypes (37493)
  • Performance improvement in .DataFrameGroupBy.all, .DataFrameGroupBy.any, .SeriesGroupBy.all, and .SeriesGroupBy.any for object dtype (50623)
  • Performance improvement in MultiIndex.argsort and MultiIndex.sort_values (48406)
  • Performance improvement in MultiIndex.size (48723)
  • Performance improvement in MultiIndex.union without missing values and without duplicates (48505, 48752)
  • Performance improvement in MultiIndex.difference (48606)
  • Performance improvement in MultiIndex set operations with sort=None (49010)
  • Performance improvement in .DataFrameGroupBy.mean, .SeriesGroupBy.mean, .DataFrameGroupBy.var, and .SeriesGroupBy.var for extension array dtypes (37493)
  • Performance improvement in MultiIndex.isin when level=None (48622, 49577)
  • Performance improvement in MultiIndex.putmask (49830)
  • Performance improvement in Index.union and MultiIndex.union when index contains duplicates (48900)
  • Performance improvement in Series.rank for pyarrow-backed dtypes (50264)
  • Performance improvement in Series.searchsorted for pyarrow-backed dtypes (50447)
  • Performance improvement in Series.fillna for extension array dtypes (49722, 50078)
  • Performance improvement in Index.join, Index.intersection and Index.union for masked dtypes when Index is monotonic (50310)
  • Performance improvement for Series.value_counts with nullable dtype (48338)
  • Performance improvement for Series constructor passing integer numpy array with nullable dtype (48338)
  • Performance improvement for DatetimeIndex constructor passing a list (48609)
  • Performance improvement in merge and DataFrame.join when joining on a sorted MultiIndex (48504)
  • Performance improvement in to_datetime when parsing strings with timezone offsets (50107)
  • Performance improvement in DataFrame.loc and Series.loc for tuple-based indexing of a MultiIndex (48384)
  • Performance improvement for MultiIndex.unique (48335)
  • Performance improvement for concat with extension array backed indexes (49128, 49178)
  • Reduce memory usage of DataFrame.to_pickle/Series.to_pickle when using BZ2 or LZMA (49068)
  • Performance improvement for ~arrays.StringArray constructor passing a numpy array with type np.str_ (49109)
  • Performance improvement in ~arrays.IntervalArray.from_tuples (50620)
  • Performance improvement in ~arrays.ArrowExtensionArray.factorize (49177)
  • Performance improvement in ~arrays.ArrowExtensionArray.__setitem__ when key is a null slice (50248)
  • Performance improvement in ~arrays.ArrowExtensionArray comparison methods when array contains NA (50524)
  • Performance improvement in ~arrays.ArrowExtensionArray.to_numpy (49973)
  • Performance improvement when parsing strings to BooleanDtype (50613)
  • Performance improvement in DataFrame.join when joining on a subset of a MultiIndex (48611)
  • Performance improvement for MultiIndex.intersection (48604)
  • Performance improvement in DataFrame.__setitem__ (46267)
  • Performance improvement in var and std for nullable dtypes (48379).
  • Performance improvement when iterating over pyarrow and nullable dtypes (49825, 49851)
  • Performance improvements to read_sas (47403, 47405, 47656, 48502)
  • Memory improvement in RangeIndex.sort_values (48801)
  • Performance improvement in Series.to_numpy if copy=True by avoiding copying twice (24345)
  • Performance improvement in DataFrameGroupBy and SeriesGroupBy when by is a categorical type and sort=False (48976)
  • Performance improvement in DataFrameGroupBy and SeriesGroupBy when by is a categorical type and observed=False (49596)
  • Performance improvement in read_stata with parameter index_col set to None (the default). Now the index will be a RangeIndex instead of Int64Index (49745)
  • Performance improvement in merge when not merging on the index - the new index will now be RangeIndex instead of Int64Index (49478)
  • Performance improvement in DataFrame.to_dict and Series.to_dict when using any non-object dtypes (46470)
  • Performance improvement in read_html when there are multiple tables (49929)
  • Performance improvement in to_datetime when using '%Y%m%d' format (17410)
  • Performance improvement in to_datetime when format is given or can be inferred (50465)
  • Performance improvement in read_csv when passing to_datetime lambda-function to date_parser and inputs have mixed timezone offsetes (35296)
  • Performance improvement in .SeriesGroupBy.value_counts with categorical dtype (46202)
  • Fixed a reference leak in read_hdf (37441)

Bug fixes

Categorical

  • Bug in Categorical.set_categories losing dtype information (48812)
  • Bug in DataFrame.groupby and Series.groupby would reorder categories when used as a grouper (48749)
  • Bug in Categorical constructor when constructing from a Categorical object and dtype="category" losing ordered-ness (49309)

Datetimelike

  • Bug in pandas.infer_freq, raising TypeError when inferred on RangeIndex (47084)
  • Bug in to_datetime incorrectly raising OverflowError with string arguments corresponding to large integers (50533)
  • Bug in to_datetime was raising on invalid offsets with errors='coerce' and infer_datetime_format=True (48633)
  • Bug in DatetimeIndex constructor failing to raise when tz=None is explicitly specified in conjunction with timezone-aware dtype or data (48659)
  • Bug in subtracting a datetime scalar from DatetimeIndex failing to retain the original freq attribute (48818)
  • Bug in pandas.tseries.holiday.Holiday where a half-open date interval causes inconsistent return types from USFederalHolidayCalendar.holidays (49075)
  • Bug in rendering DatetimeIndex and Series and DataFrame with timezone-aware dtypes with dateutil or zoneinfo timezones near daylight-savings transitions (49684)
  • Bug in to_datetime was raising ValueError when parsing Timestamp, datetime.datetime, datetime.date, or np.datetime64 objects when non-ISO8601 format was passed (49298, 50036)
  • Bug in to_datetime was raising ValueError when parsing empty string and non-ISO8601 format was passed. Now, empty strings will be parsed as NaT, for compatibility with how is done for ISO8601 formats (50251)
  • Bug in Timestamp was showing UserWarning, which was not actionable by users, when parsing non-ISO8601 delimited date strings (50232)
  • Bug in to_datetime was showing misleading ValueError when parsing dates with format containing ISO week directive and ISO weekday directive (50308)
  • Bug in Timestamp.round when the freq argument has zero-duration (e.g. "0ns") returning incorrect results instead of raising (49737)
  • Bug in to_datetime was not raising ValueError when invalid format was passed and errors was 'ignore' or 'coerce' (50266)
  • Bug in DateOffset was throwing TypeError when constructing with milliseconds and another super-daily argument (49897)
  • Bug in to_datetime was not raising ValueError when parsing string with decimal date with format '%Y%m%d' (50051)
  • Bug in to_datetime was not converting None to NaT when parsing mixed-offset date strings with ISO8601 format (50071)
  • Bug in to_datetime was not returning input when parsing out-of-bounds date string with errors='ignore' and format='%Y%m%d' (14487)
  • Bug in to_datetime was converting timezone-naive datetime.datetime to timezone-aware when parsing with timezone-aware strings, ISO8601 format, and utc=False (50254)
  • Bug in to_datetime was throwing ValueError when parsing dates with ISO8601 format where some values were not zero-padded (21422)
  • Bug in to_datetime was giving incorrect results when using format='%Y%m%d' and errors='ignore' (26493)
  • Bug in to_datetime was failing to parse date strings 'today' and 'now' if format was not ISO8601 (50359)
  • Bug in Timestamp.utctimetuple raising a TypeError (32174)
  • Bug in to_datetime was raising ValueError when parsing mixed-offset Timestamp with errors='ignore' (50585)

Timedelta

  • Bug in to_timedelta raising error when input has nullable dtype Float64 (48796)
  • Bug in Timedelta constructor incorrectly raising instead of returning NaT when given a np.timedelta64("nat") (48898)
  • Bug in Timedelta constructor failing to raise when passed both a Timedelta object and keywords (e.g. days, seconds) (48898)

Timezones

  • Bug in Series.astype and DataFrame.astype with object-dtype containing multiple timezone-aware datetime objects with heterogeneous timezones to a DatetimeTZDtype incorrectly raising (32581)
  • Bug in to_datetime was failing to parse date strings with timezone name when format was specified with %Z (49748)
  • Better error message when passing invalid values to ambiguous parameter in Timestamp.tz_localize (49565)
  • Bug in string parsing incorrectly allowing a Timestamp to be constructed with an invalid timezone, which would raise when trying to print (50668)

Numeric

  • Bug in DataFrame.add cannot apply ufunc when inputs contain mixed DataFrame type and Series type (39853)
  • Bug in arithmetic operations on Series not propagating mask when combining masked dtypes and numpy dtypes (45810, 42630)
  • Bug in DataFrame reduction methods (e.g. DataFrame.sum) with object dtype, axis=1 and numeric_only=False would not be coerced to float (49551)
  • Bug in DataFrame.sem and Series.sem where an erroneous TypeError would always raise when using data backed by an ArrowDtype (49759)
  • Bug in Series.__add__ casting to object for list and masked Series (22962)

Conversion

  • Bug in constructing Series with int64 dtype from a string list raising instead of casting (44923)
  • Bug in constructing Series with masked dtype and boolean values with NA raising (42137)
  • Bug in DataFrame.eval incorrectly raising an AttributeError when there are negative values in function call (46471)
  • Bug in Series.convert_dtypes not converting dtype to nullable dtype when Series contains NA and has dtype object (48791)
  • Bug where any ExtensionDtype subclass with kind="M" would be interpreted as a timezone type (34986)
  • Bug in .arrays.ArrowExtensionArray that would raise NotImplementedError when passed a sequence of strings or binary (49172)
  • Bug in Series.astype raising pyarrow.ArrowInvalid when converting from a non-pyarrow string dtype to a pyarrow numeric type (50430)
  • Bug in Series.to_numpy converting to NumPy array before applying na_value (48951)
  • Bug in to_datetime was not respecting exact argument when format was an ISO8601 format (12649)
  • Bug in TimedeltaArray.astype raising TypeError when converting to a pyarrow duration type (49795)

Strings

  • Bug in pandas.api.dtypes.is_string_dtype that would not return True for StringDtype (15585)
  • Bug in converting string dtypes to "datetime64[ns]" or "timedelta64[ns]" incorrectly raising TypeError (36153)

Interval

  • Bug in IntervalIndex.is_overlapping incorrect output if interval has duplicate left boundaries (49581)
  • Bug in Series.infer_objects failing to infer IntervalDtype for an object series of Interval objects (50090)

Indexing

  • Bug in DataFrame.__setitem__ raising when indexer is a DataFrame with boolean dtype (47125)
  • Bug in DataFrame.reindex filling with wrong values when indexing columns and index for uint dtypes (48184)
  • Bug in DataFrame.loc when setting DataFrame with different dtypes coercing values to single dtype (50467)
  • Bug in DataFrame.sort_values where None was not returned when by is empty list and inplace=True (50643)
  • Bug in DataFrame.loc coercing dtypes when setting values with a list indexer (49159)
  • Bug in Series.loc raising error for out of bounds end of slice indexer (50161)
  • Bug in DataFrame.loc raising ValueError with bool indexer and MultiIndex (47687)
  • Bug in DataFrame.loc raising IndexError when setting values for a pyarrow-backed column with a non-scalar indexer (50085)
  • Bug in DataFrame.__setitem__ raising ValueError when right hand side is DataFrame with MultiIndex columns (49121)
  • Bug in DataFrame.reindex casting dtype to object when DataFrame has single extension array column when re-indexing columns and index (48190)
  • Bug in DataFrame.iloc raising IndexError when indexer is a Series with numeric extension array dtype (49521)
  • Bug in ~DataFrame.describe when formatting percentiles in the resulting index showed more decimals than needed (46362)
  • Bug in DataFrame.compare does not recognize differences when comparing NA with value in nullable dtypes (48939)
  • Bug in DataFrame.isetitem coercing extension array dtypes in DataFrame to object (49922)
  • Bug in BusinessHour would cause creation of DatetimeIndex to fail when no opening hour was included in the index (49835)

Missing

  • Bug in Index.equals raising TypeError when Index consists of tuples that contain NA (48446)
  • Bug in Series.map caused incorrect result when data has NaNs and defaultdict mapping was used (48813)
  • Bug in NA raising a TypeError instead of return NA when performing a binary operation with a bytes object (49108)
  • Bug in DataFrame.update with overwrite=False raising TypeError when self has column with NaT values and column not present in other (16713)
  • Bug in Series.replace raising RecursionError when replacing value in object-dtype Series containing NA (47480)
  • Bug in Series.replace raising RecursionError when replacing value in numeric Series with NA (50758)

MultiIndex

  • Bug in MultiIndex.get_indexer not matching NaN values (29252, 37222, 38623, 42883, 43222, 46173, 48905)
  • Bug in MultiIndex.argsort raising TypeError when index contains NA (48495)
  • Bug in MultiIndex.difference losing extension array dtype (48606)
  • Bug in MultiIndex.set_levels raising IndexError when setting empty level (48636)
  • Bug in MultiIndex.unique losing extension array dtype (48335)
  • Bug in MultiIndex.intersection losing extension array (48604)
  • Bug in MultiIndex.union losing extension array (48498, 48505, 48900)
  • Bug in MultiIndex.union not sorting when sort=None and index contains missing values (49010)
  • Bug in MultiIndex.append not checking names for equality (48288)
  • Bug in MultiIndex.symmetric_difference losing extension array (48607)
  • Bug in MultiIndex.join losing dtypes when MultiIndex has duplicates (49830)
  • Bug in MultiIndex.putmask losing extension array (49830)
  • Bug in MultiIndex.value_counts returning a Series indexed by flat index of tuples instead of a MultiIndex (49558)

I/O

  • Bug in read_sas caused fragmentation of DataFrame and raised .errors.PerformanceWarning (48595)
  • Improved error message in read_excel by including the offending sheet name when an exception is raised while reading a file (48706)
  • Bug when a pickling a subset PyArrow-backed data that would serialize the entire data instead of the subset (42600)
  • Bug in read_sql_query ignoring dtype argument when chunksize is specified and result is empty (50245)
  • Bug in read_csv for a single-line csv with fewer columns than names raised .errors.ParserError with engine="c" (47566)
  • Bug in read_json raising with orient="table" and NA value (40255)
  • Bug in displaying string dtypes not showing storage option (50099)
  • Bug in DataFrame.to_string with header=False that printed the index name on the same line as the first row of the data (49230)
  • Bug in DataFrame.to_string ignoring float formatter for extension arrays (39336)
  • Fixed memory leak which stemmed from the initialization of the internal JSON module (49222)
  • Fixed issue where json_normalize would incorrectly remove leading characters from column names that matched the sep argument (49861)
  • Bug in DataFrame.to_dict not converting NA to None (50795)
  • Bug in DataFrame.to_json where it would segfault when failing to encode a string (50307)

Period

  • Bug in Period.strftime and PeriodIndex.strftime, raising UnicodeDecodeError when a locale-specific directive was passed (46319)
  • Bug in adding a Period object to an array of DateOffset objects incorrectly raising TypeError (50162)
  • Bug in Period where passing a string with finer resolution than nanosecond would result in a KeyError instead of dropping the extra precision (50417)

Plotting

  • Bug in DataFrame.plot.hist, not dropping elements of weights corresponding to NaN values in data (48884)
  • ax.set_xlim was sometimes raising UserWarning which users couldn't address due to set_xlim not accepting parsing arguments - the converter now uses Timestamp instead (49148)

Groupby/resample/rolling

  • Bug in .ExponentialMovingWindow with online not raising a NotImplementedError for unsupported operations (48834)
  • Bug in DataFrameGroupBy.sample raises ValueError when the object is empty (48459)
  • Bug in Series.groupby raises ValueError when an entry of the index is equal to the name of the index (48567)
  • Bug in DataFrameGroupBy.resample produces inconsistent results when passing empty DataFrame (47705)
  • Bug in .DataFrameGroupBy and .SeriesGroupBy would not include unobserved categories in result when grouping by categorical indexes (49354)
  • Bug in .DataFrameGroupBy and .SeriesGroupBy would change result order depending on the input index when grouping by categoricals (49223)
  • Bug in .DataFrameGroupBy and .SeriesGroupBy when grouping on categorical data would sort result values even when used with sort=False (42482)
  • Bug in .DataFrameGroupBy.apply and SeriesGroupBy.apply with as_index=False would not attempt the computation without using the grouping keys when using them failed with a TypeError (49256)
  • Bug in .DataFrameGroupBy.describe would describe the group keys (49256)
  • Bug in .SeriesGroupBy.describe with as_index=False would have the incorrect shape (49256)
  • Bug in .DataFrameGroupBy and .SeriesGroupBy with dropna=False would drop NA values when the grouper was categorical (36327)
  • Bug in .SeriesGroupBy.nunique would incorrectly raise when the grouper was an empty categorical and observed=True (21334)
  • Bug in .SeriesGroupBy.nth would raise when grouper contained NA values after subsetting from a DataFrameGroupBy (26454)
  • Bug in DataFrame.groupby would not include a .Grouper specified by key in the result when as_index=False (50413)
  • Bug in .DataFrameGrouBy.value_counts would raise when used with a .TimeGrouper (50486)
  • Bug in Resampler.size caused a wide DataFrame to be returned instead of a Series with MultiIndex (46826)
  • Bug in .DataFrameGroupBy.transform and .SeriesGroupBy.transform would raise incorrectly when grouper had axis=1 for "idxmin" and "idxmax" arguments (45986)
  • Bug in .DataFrameGroupBy would raise when used with an empty DataFrame, categorical grouper, and dropna=False (50634)
  • Bug in .SeriesGroupBy.value_counts did not respect sort=False (50482)

Reshaping

  • Bug in DataFrame.pivot_table raising TypeError for nullable dtype and margins=True (48681)
  • Bug in DataFrame.unstack and Series.unstack unstacking wrong level of MultiIndex when MultiIndex has mixed names (48763)
  • Bug in DataFrame.melt losing extension array dtype (41570)
  • Bug in DataFrame.pivot not respecting None as column name (48293)
  • Bug in join when left_on or right_on is or includes a CategoricalIndex incorrectly raising AttributeError (48464)
  • Bug in DataFrame.pivot_table raising ValueError with parameter margins=True when result is an empty DataFrame (49240)
  • Clarified error message in merge when passing invalid validate option (49417)
  • Bug in DataFrame.explode raising ValueError on multiple columns with NaN values or empty lists (46084)
  • Bug in DataFrame.transpose with IntervalDtype column with timedelta64[ns] endpoints (44917)

Sparse

  • Bug in Series.astype when converting a SparseDtype with datetime64[ns] subtype to int64 dtype raising, inconsistent with the non-sparse behavior (49631,50087)
  • Bug in Series.astype when converting a from datetime64[ns] to Sparse[datetime64[ns]] incorrectly raising (50082)

ExtensionArray

  • Bug in Series.mean overflowing unnecessarily with nullable integers (48378)
  • Bug in Series.tolist for nullable dtypes returning numpy scalars instead of python scalars (49890)
  • Bug in Series.round for pyarrow-backed dtypes raising AttributeError (50437)
  • Bug when concatenating an empty DataFrame with an ExtensionDtype to another DataFrame with the same ExtensionDtype, the resulting dtype turned into object (48510)
  • Bug in array.PandasArray.to_numpy raising with NA value when na_value is specified (40638)
  • Bug in api.types.is_numeric_dtype where a custom ExtensionDtype would not return True if _is_numeric returned True (50563)
  • Bug in api.types.is_integer_dtype, api.types.is_unsigned_integer_dtype, api.types.is_signed_integer_dtype, api.types.is_float_dtype where a custom ExtensionDtype would not return True if kind returned the corresponding NumPy type (50667)

Styler

  • Fix ~pandas.io.formats.style.Styler.background_gradient for nullable dtype Series with NA values (50712)

Metadata

  • Fixed metadata propagation in DataFrame.corr and DataFrame.cov (28283)

Other

  • Bug in Series.searchsorted inconsistent behavior when accepting DataFrame as parameter value (49620)

Contributors