DEPR: error_bad_lines and warn_bad_lines for read_csv #40413

lithomas1 · 2021-03-13T04:56:48Z

closes API/ENH: read_csv handling of bad lines (too many/few fields) #15122
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

Summary of contents
- Adds new on_bad_lines parameter (I found on_bad_lines a more explicit name than bad_lines)
- This defaults to None at first in order to preserve compatibility, however it should be changed to error in 2.0 after
error_bad_lines and warn_bad_lines are removed.
- Cleanup of some C/Python Parser code ( add enum instead of using 2 variables for C and only use on_bad_lines in Python)

pep8speaks · 2021-03-16T23:14:38Z

Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-05-28 00:23:35 UTC

…pr-bad-lines

jreback

cool, am +1 on this change, some comments

doc/source/user_guide/io.rst

pandas/_libs/parsers.pyx

pandas/io/parsers/python_parser.py

pandas/io/parsers/readers.py

jreback · 2021-04-02T02:01:33Z

pandas/io/parsers/readers.py

@@ -382,6 +402,7 @@
    "memory_map": False,
    "error_bad_lines": True,
    "warn_bad_lines": True,
+    "on_bad_lines": None,


yeah should remove error/warn_bad_lines from here

_get_options_with_defaults is really spaghetti-fied right now, so removing this would not the args not passed to the parser. I will try to clean up _get_options_with_defaults in a future PR if I have time.

jreback · 2021-04-02T02:02:23Z

cc @gfyoung if you'd have a look

pandas/tests/io/parser/common/test_common_basic.py

jreback

we need to set the original options to None to avoid way more complexity.

doc/source/user_guide/io.rst

pandas/_libs/parsers.pyx

pandas/io/parsers/base_parser.py

jreback · 2021-04-09T00:10:12Z

pandas/io/parsers/base_parser.py

+            else:
+                raise ValueError(f"Argument {on_bad_lines} is invalid for on_bad_lines")
+        else:
+            if kwds.get("error_bad_lines"):


these need a deprecation warning

_deprecated_defaults handles this for us.

pandas/io/parsers/readers.py

jreback

maybe i am not seeing it. but it seems that the old options (error_bad_lines / warn_bad_lines) are still passed around. these should be handled in exactly 1 place, then immediately discarded (with a possible warning / error if multiple things are specified), and then only on_bad_lines should exist.

jreback · 2021-05-06T01:47:22Z

pandas/io/parsers/base_parser.py

+            self.on_bad_lines = self.BadLineHandleMethod.SKIP
+        else:
+            raise ValueError(f"Argument {on_bad_lines} is invalid for on_bad_lines")
+        # Override on_bad_lines w/ deprecated args for backward compatibility


shouldn't all of these cases show a deprecation warning? L227-238

Like, I said before this is handled by _deprecated_defaults. Ideally _deprecated_defaults wouldn't exist, but as we all know, read_csv is very bloated in terms of API, so its nice to show the deprecation warnings from one place. If you are still unsure, this is tested in this PR here https://github.com/pandas-dev/pandas/pull/40413/files#diff-2f09043040b80b8e52cb8525b43b1653de0ce7d04e0999ca8e0dbe69b05b88faR758-R770

jreback · 2021-05-28T16:58:33Z

thanks @lithomas1 this was a bear! thanks for sticking with it

lithomas1 · 2021-05-28T20:47:05Z

Thanks for the reviews @jreback. Would it be possible to mention this in the deprecations tracker #30228?(I can't edit it myself.)

jreback · 2021-05-28T22:31:39Z

updated

… from `read_csv` & enabling more tests ### What changes were proposed in this pull request? This PR proposes to remove `squeeze` parameter from `read_csv` to follow the behavior of latest pandas. See pandas-dev/pandas#40413 and pandas-dev/pandas#43427 for detail. This PR also enables more tests for pandas 2.0.0 and above. ### Why are the changes needed? To follow the behavior of latest pandas, and increase the test coverage. ### Does this PR introduce _any_ user-facing change? `squeeze` will be no longer available from `read_csv`. Otherwise, it's test-only. ### How was this patch tested? Enabling & updating the existing tests. Closes #42551 from itholic/pandas_remaining_tests. Authored-by: itholic <haejoon.lee@databricks.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

… from `read_csv` & enabling more tests ### What changes were proposed in this pull request? This PR proposes to remove `squeeze` parameter from `read_csv` to follow the behavior of latest pandas. See pandas-dev/pandas#40413 and pandas-dev/pandas#43427 for detail. This PR also enables more tests for pandas 2.0.0 and above. ### Why are the changes needed? To follow the behavior of latest pandas, and increase the test coverage. ### Does this PR introduce _any_ user-facing change? `squeeze` will be no longer available from `read_csv`. Otherwise, it's test-only. ### How was this patch tested? Enabling & updating the existing tests. Closes apache#42551 from itholic/pandas_remaining_tests. Authored-by: itholic <haejoon.lee@databricks.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

lithomas1 added 2 commits March 12, 2021 20:55

Deprecate

4e867e9

stacklevel/warnings fixes

d230035

lithomas1 added Deprecate Functionality to remove in pandas IO CSV read_csv, to_csv labels Mar 13, 2021

Fixes

ce5bf29

lithomas1 requested review from jreback and jorisvandenbossche March 13, 2021 17:39

lithomas1 added 2 commits March 13, 2021 10:39

Fix tests

8629f87

Doc fixes for green

06f87a1

lithomas1 requested a review from WillAyd March 14, 2021 17:21

Merge branch 'master' into depr-bad-lines

edeef7e

lithomas1 requested a review from gfyoung March 16, 2021 23:13

Merge branch 'master' into depr-bad-lines

f806a4d

lithomas1 added 2 commits March 16, 2021 16:15

Update test_common_basic.py

5b08a88

Update test_common_basic.py

af3fd15

lithomas1 requested review from jreback and removed request for jreback March 22, 2021 16:08

lithomas1 and others added 2 commits March 28, 2021 11:23

Merge branch 'master' into depr-bad-lines

f70f34e

Merge branch 'master' of https://github.com/pandas-dev/pandas into de…

0c76180

…pr-bad-lines

jreback requested changes Apr 2, 2021

View reviewed changes

gfyoung reviewed Apr 2, 2021

View reviewed changes

pandas/tests/io/parser/common/test_common_basic.py Outdated Show resolved Hide resolved

lithomas1 added 2 commits April 5, 2021 19:20

Address Code Review

a0406b5

oops

89fdc70

lithomas1 requested review from gfyoung and jreback April 6, 2021 03:37

jreback requested changes Apr 9, 2021

View reviewed changes

lithomas1 and others added 2 commits April 27, 2021 15:03

Merge branch 'master' into depr-bad-lines

f7265a3

Address code review

1e20b53

jreback requested changes May 6, 2021

View reviewed changes

lithomas1 and others added 7 commits May 22, 2021 19:52

Consolidate

e267aa4

Clarify behavior

e724d0b

Merge branch 'master' into depr-bad-lines

fdef68e

Merge branch 'master' into depr-bad-lines

a220293

typing

9b8468a

Fix failed test

a6af9aa

Clean code

2f70edc

lithomas1 requested a review from jreback May 25, 2021 00:17

lithomas1 added this to the 1.3 milestone May 25, 2021

lithomas1 and others added 4 commits May 27, 2021 16:04

Merge branch 'master' into depr-bad-lines

93c37df

Update v1.3.0.rst

cf3201c

Update readers.py

4911b27

Fix stacklevel

f28316b

jreback approved these changes May 28, 2021

View reviewed changes

jreback merged commit fd346ae into pandas-dev:master May 28, 2021

lithomas1 deleted the depr-bad-lines branch May 28, 2021 20:36

jreback mentioned this pull request May 28, 2021

DEPR: log of deprecations in 1.x (to be removed in 2.0) #30228

Closed

TLouf pushed a commit to TLouf/pandas that referenced this pull request Jun 1, 2021

DEPR: error_bad_lines and warn_bad_lines for read_csv (pandas-dev#40413)

fb6131d

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021

DEPR: error_bad_lines and warn_bad_lines for read_csv (pandas-dev#40413)

0e51d09

phofl mentioned this pull request Sep 29, 2022

DEP: Enforce deprecations of read_csv keywords #48849

Merged

LuighiV mentioned this pull request Apr 6, 2023

error_bad_lines argument has been deprecated simonw/csvs-to-sqlite#88

Open

itholic mentioned this pull request Aug 18, 2023

[SPARK-43563][SPARK-43459][SPARK-43451][SPARK-43506] Remove squeeze from read_csv & enabling more tests. apache/spark#42551

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: error_bad_lines and warn_bad_lines for read_csv #40413

DEPR: error_bad_lines and warn_bad_lines for read_csv #40413

lithomas1 commented Mar 13, 2021 •

edited

pep8speaks commented Mar 16, 2021 •

edited

jreback left a comment

jreback Apr 2, 2021

lithomas1 Apr 6, 2021 •

edited

jreback commented Apr 2, 2021

jreback left a comment

jreback Apr 9, 2021

lithomas1 Apr 28, 2021

jreback left a comment

jreback May 6, 2021

lithomas1 May 23, 2021

jreback commented May 28, 2021

lithomas1 commented May 28, 2021

jreback commented May 28, 2021

DEPR: error_bad_lines and warn_bad_lines for read_csv #40413

DEPR: error_bad_lines and warn_bad_lines for read_csv #40413

Conversation

lithomas1 commented Mar 13, 2021 • edited

pep8speaks commented Mar 16, 2021 • edited

Comment last updated at 2021-05-28 00:23:35 UTC

jreback left a comment

Choose a reason for hiding this comment

jreback Apr 2, 2021

Choose a reason for hiding this comment

lithomas1 Apr 6, 2021 • edited

Choose a reason for hiding this comment

jreback commented Apr 2, 2021

jreback left a comment

Choose a reason for hiding this comment

jreback Apr 9, 2021

Choose a reason for hiding this comment

lithomas1 Apr 28, 2021

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback May 6, 2021

Choose a reason for hiding this comment

lithomas1 May 23, 2021

Choose a reason for hiding this comment

jreback commented May 28, 2021

lithomas1 commented May 28, 2021

jreback commented May 28, 2021

lithomas1 commented Mar 13, 2021 •

edited

pep8speaks commented Mar 16, 2021 •

edited

lithomas1 Apr 6, 2021 •

edited