API: Deprecate regex=True default in Series.str.replace #36695

dsaxton · 2020-09-28T00:58:54Z

From some old discussion it looks like there was interest in changing the default value of regex from True to False within Series.str.replace. I think this makes sense since it would align this method with others within pandas along with the standard library, and would also make fixing #24804 slightly less disruptive. I'm assuming this would warrant a deprecation note in the next whatsnew?

I think this part of the docs will need to be updated: https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html#splitting-and-replacing-strings

#24809

jorisvandenbossche · 2020-09-28T18:43:12Z

There are many tests that are using this (and are now raising a warning) that would need updating.

While I agree that a default of regex=False would probably be better ideally, warning about this in all cases also seems quite annoying ..
Would it somehow be possible to detect in advance if the pattern is regex-like or rather a plain string (in which case replacing it with regex=False would not give a different result). Probably not that straightforward given the complexity of regexes .. (but maybe some simple cases, like only characters, could be detected)

dsaxton · 2020-09-28T19:18:15Z

There are many tests that are using this (and are now raising a warning) that would need updating.

While I agree that a default of regex=False would probably be better ideally, warning about this in all cases also seems quite annoying ..
Would it somehow be possible to detect in advance if the pattern is regex-like or rather a plain string (in which case replacing it with regex=False would not give a different result). Probably not that straightforward given the complexity of regexes .. (but maybe some simple cases, like only characters, could be detected)

Yeah this is a good point, I think looking for regex special characters and only then warning could make the most sense

…ault

doc/source/user_guide/text.rst

…ault

jorisvandenbossche

Can you also add a whatsnew note?

pandas/core/strings.py

doc/source/user_guide/text.rst

dsaxton · 2020-09-29T12:25:13Z

Can you also add a whatsnew note?

Would this be an update for 1.1.4 or 1.2?

…ault

dsaxton · 2020-10-07T15:06:13Z

yes, you are excluding them from the warning now though; can we provide a more specific warning message for single char regexes?

Added an additional warning for single character regexes:

[ins] In [3]: s.str.replace(".", "")
<ipython-input-3-ee1a585145f8>:1: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.
  s.str.replace(".", "")
Out[3]: 
0    a
1    b
dtype: object

[ins] In [4]: s.str.replace("^.", "")
<ipython-input-4-3940e91d28a6>:1: FutureWarning: The default value of regex will change from True to False in a future version.
  s.str.replace("^.", "")
Out[4]: 
0    
1    
dtype: object

dsaxton · 2020-10-07T15:08:08Z

looks good. doc comment & pls add a deprecation note in the whatsnew (also add to the deprecation master issue) #30228

Updated the OP in that issue

…ault

jreback · 2020-10-10T15:39:23Z

doc/source/user_guide/text.rst

-Some caution must be taken to keep regular expressions in mind! For example, the
-following code will cause trouble because of the regular expression meaning of
-``$``:
+Some caution must be taken when dealing with regular expressions! The current behavior


can you do this

jreback · 2020-10-10T15:39:36Z

doc/source/user_guide/text.rst

-Some caution must be taken to keep regular expressions in mind! For example, the
-following code will cause trouble because of the regular expression meaning of
-``$``:
+Some caution must be taken when dealing with regular expressions! The current behavior


also add a versionchanged tag

doc/source/user_guide/text.rst

jreback · 2020-10-10T15:41:44Z

pandas/tests/series/methods/test_replace.py

+
+    def test_str_replace_regex_default_raises_warning(self):
+        # https://github.com/pandas-dev/pandas/pull/24809
+        s = pd.Series(["a", "b", "c"])


can you check the messages on this warning

jreback · 2020-10-10T15:42:11Z

pandas/core/strings/accessor.py

+            if isinstance(pat, str) and any(c in pat for c in ".+*|^$?[](){}\\"):
+                # warn only in cases where regex behavior would differ from literal
+                msg = (
+                    "The default value of regex will change from True to False "


an you make sure that we are testing both of these warnings (as i see you only added a single test)

…ault

jreback · 2020-10-10T16:30:57Z

doc/source/user_guide/text.rst

-Some caution must be taken to keep regular expressions in mind! For example, the
-following code will cause trouble because of the regular expression meaning of
-``$``:
+Some caution must be taken when dealing with regular expressions! The current behavior


no i mean a ..note:: a sphinx-note (which puts a highlite box around this)

jreback · 2020-10-10T16:31:26Z

doc/source/user_guide/text.rst

-Some caution must be taken to keep regular expressions in mind! For example, the
-following code will cause trouble because of the regular expression meaning of
-``$``:
+Some caution must be taken when dealing with regular expressions! The current behavior


we could also do a ..warning:: which is a red-box and more prominent, but either way want to call out this

doc/source/user_guide/text.rst

…ault

jreback

very nice

jreback · 2020-10-10T22:21:51Z

doc/source/user_guide/text.rst

-
-   # This does what you'd naively expect:
-   dollars.str.replace("$", "")
+.. warning::


jreback · 2020-10-10T22:23:29Z

merge on most builds green (some are giving issues now).

…6695)

Daniel Saxton added 4 commits September 27, 2020 19:38

API: Deprecate regex=True default in Series.str.replace

52f6d78

Format

3b9bcb2

Stacklevel

c51e837

Test

3761625

dsaxton added replace replace method API - Consistency Internal Consistency of API/Behavior labels Sep 28, 2020

Daniel Saxton added 3 commits September 27, 2020 20:00

Test name

7583b7f

Link discussion

8e0d1a5

Set regex

79e4400

Daniel Saxton added 7 commits September 28, 2020 16:22

Update

ecc5786

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

cf659b0

…ault

Update

628cfa3

Fix

f216022

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

e6c79a3

…ault

Check for str

e6b3e0f

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

e3ce080

…ault

dsaxton commented Sep 28, 2020

View reviewed changes

doc/source/user_guide/text.rst Outdated Show resolved Hide resolved

Daniel Saxton added 6 commits September 28, 2020 19:06

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

752d322

…ault

Doc updates

ee67284

Nit

c832a21

Silence some warnings

09b07e6

Edit

c8541da

Edit

bd31857

jorisvandenbossche reviewed Sep 29, 2020

View reviewed changes

pandas/core/strings.py Outdated Show resolved Hide resolved

doc/source/user_guide/text.rst Outdated Show resolved Hide resolved

Daniel Saxton added 2 commits September 29, 2020 07:21

Question mark

377a2ba

Update doc

d6b4fbe

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

84dfe71

…ault

Daniel Saxton added 2 commits October 7, 2020 09:43

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

cd18347

…ault

Update warning

f872011

Daniel Saxton added 6 commits October 7, 2020 10:56

rst lint

c0a473e

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

727986e

…ault

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

f49f778

…ault

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

d3d155a

…ault

Nit + emphasis

89013db

regex=False

e78017c

jreback requested changes Oct 10, 2020

View reviewed changes

Daniel Saxton added 3 commits October 10, 2020 11:10

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

0150b2b

…ault

Update

e396ce5

fix

e799b12

jreback requested changes Oct 10, 2020

View reviewed changes

Daniel Saxton added 3 commits October 10, 2020 15:45

rst fix

9f7545f

Merge remote-tracking branch 'upstream/master' into replace-regex-def…

6be90a4

…ault

Make a warning

8a4a833

jreback approved these changes Oct 10, 2020

View reviewed changes

doc/source/user_guide/text.rst

# This does what you'd naively expect:

dollars.str.replace("$", "")

.. warning::

Copy link

Contributor

jreback Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great

dsaxton merged commit 1b9641c into pandas-dev:master Oct 10, 2020

dsaxton deleted the replace-regex-default branch October 10, 2020 23:01

kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020

API: Deprecate regex=True default in Series.str.replace (pandas-dev#3…

397252f

…6695)

danlamanna mentioned this pull request Dec 30, 2020

Explicitly set regex=True in pandas str.replace ImageMarkup/isic-challenge-scoring#117

Merged

rhshadrach mentioned this pull request May 9, 2021

DOC: specify regex=True in str.replace #41397

Merged

1 task

simonjayhawkins mentioned this pull request May 21, 2021

str.replace('.','') should replace every character? #24804

Closed

dfukagaw28 mentioned this pull request May 17, 2022

FutureWarning: The default value of regex will change from True to False in a future version. ducis28/paiza-tools#1

Closed

mroeschke mentioned this pull request Nov 2, 2022

DEPR: Change str.replace(regex) from True to False & single behavior #49486

Merged

1 task

SorooshMani-NOAA mentioned this pull request Apr 21, 2023

Fix for compatibility with ADCIRC GAHM oceanmodeling/StormEvents#76

Merged

abretaud mentioned this pull request Apr 26, 2023

Fix compatibility issue with pandas >=2 nextgenusfs/funannotate#906

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: Deprecate regex=True default in Series.str.replace #36695

API: Deprecate regex=True default in Series.str.replace #36695

dsaxton commented Sep 28, 2020 •

edited

jorisvandenbossche commented Sep 28, 2020

dsaxton commented Sep 28, 2020

jorisvandenbossche left a comment

dsaxton commented Sep 29, 2020

dsaxton commented Oct 7, 2020 •

edited

dsaxton commented Oct 7, 2020

jreback Oct 10, 2020

jreback Oct 10, 2020

jreback Oct 10, 2020

jreback Oct 10, 2020

jreback Oct 10, 2020

jreback Oct 10, 2020

jreback left a comment

jreback Oct 10, 2020

jreback commented Oct 10, 2020

API: Deprecate regex=True default in Series.str.replace #36695

API: Deprecate regex=True default in Series.str.replace #36695

Conversation

dsaxton commented Sep 28, 2020 • edited

jorisvandenbossche commented Sep 28, 2020

dsaxton commented Sep 28, 2020

jorisvandenbossche left a comment

Choose a reason for hiding this comment

dsaxton commented Sep 29, 2020

dsaxton commented Oct 7, 2020 • edited

dsaxton commented Oct 7, 2020

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback Oct 10, 2020

Choose a reason for hiding this comment

jreback commented Oct 10, 2020

dsaxton commented Sep 28, 2020 •

edited

dsaxton commented Oct 7, 2020 •

edited