Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Review rm_rf handling of FileNotFoundErrors #6044

Merged
merged 1 commit into from
Oct 23, 2019
Merged

Review rm_rf handling of FileNotFoundErrors #6044

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/6044.bugfix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Properly ignore ``FileNotFoundError`` exceptions when trying to remove old temporary directories,
for instance when multiple processes try to remove the same directory (common with ``pytest-xdist``
for example).
29 changes: 22 additions & 7 deletions src/_pytest/pathlib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,21 +38,35 @@ def ensure_reset_dir(path):
path.mkdir()


def on_rm_rf_error(func, path: str, exc, *, start_path):
"""Handles known read-only errors during rmtree."""
excvalue = exc[1]
def on_rm_rf_error(func, path: str, exc, *, start_path) -> bool:
"""Handles known read-only errors during rmtree.

The returned value is used only by our own tests.
"""
exctype, excvalue = exc[:2]

# another process removed the file in the middle of the "rm_rf" (xdist for example)
# more context: https://github.com/pytest-dev/pytest/issues/5974#issuecomment-543799018
if isinstance(excvalue, FileNotFoundError):
return False

if not isinstance(excvalue, PermissionError):
warnings.warn(
PytestWarning("(rm_rf) error removing {}: {}".format(path, excvalue))
PytestWarning(
"(rm_rf) error removing {}\n{}: {}".format(path, exctype, excvalue)
)
)
return
return False

if func not in (os.rmdir, os.remove, os.unlink):
warnings.warn(
PytestWarning("(rm_rf) error removing {}: {}".format(path, excvalue))
PytestWarning(
"(rm_rf) unknown function {} when removing {}:\n{}: {}".format(
path, func, exctype, excvalue
)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should not warn on open, should it?

============== test session starts ===============
platform linux -- Python 3.7.4, pytest-4.6.7.dev6+g46aa29ea5.d20191026, py-1.8.1.dev11+g34f716fe, pluggy-0.12.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('…/Vcs/pytest/.hypothesis/examples')
rootdir: …/Vcs/pytest, inifile: tox.ini
plugins: hypothesis-4.23.8, forked-1.0.2, repeat-0.8.0, xdist-1.30.1.dev1+g79dd52b.d20191018
collected 7 items / 5 deselected / 2 selected

testing/test_faulthandler.py EE            [100%]

===================== ERRORS =====================
______ ERROR at setup of test_timeout[True] ______
Traceback (most recent call last):
  File "/usr/lib/python3.7/shutil.py", line 426, in _rmtree_safe_fd
    dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '.pytest_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.7/shutil.py", line 428, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "…/Vcs/pytest/src/_pytest/pathlib.py", line 65, in on_rm_rf_error
    path, func, exctype, excvalue
pytest.PytestWarning: (rm_rf) unknown function /tmp/pytest-of-user/garbage-d7f65083-c870-4de6-8d7c-acf43dbafdf1/test_cache_failure_warns0/.pytest_cache when removing <built-in function open>:
<class 'PermissionError'>: [Errno 13] Permission denied: '.pytest_cache'
_____ ERROR at setup of test_timeout[False] ______
Traceback (most recent call last):
  File "/usr/lib/python3.7/shutil.py", line 426, in _rmtree_safe_fd
    dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '.pytest_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.7/shutil.py", line 428, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "…/Vcs/pytest/src/_pytest/pathlib.py", line 65, in on_rm_rf_error
    path, func, exctype, excvalue
pytest.PytestWarning: (rm_rf) unknown function /tmp/pytest-of-user/garbage-c10bc5b6-43ba-4bb0-b73d-1ab5d2bbcb1a/test_cache_failure_warns0/.pytest_cache when removing <built-in function open>:
<class 'PermissionError'>: [Errno 13] Permission denied: '.pytest_cache'
============ short test summary info =============
ERROR testing/test_faulthandler.py::test_timeout[True]
ERROR testing/test_faulthandler.py::test_timeout[False]

)
return
return False

# Chmod + retry.
import stat
Expand All @@ -73,6 +87,7 @@ def chmod_rw(p: str):
chmod_rw(str(path))

func(path)
return True


def rm_rf(path: Path):
Expand Down
4 changes: 4 additions & 0 deletions testing/test_tmpdir.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,10 @@ def test_on_rm_rf_error(self, tmp_path):
on_rm_rf_error(os.unlink, str(fn), exc_info, start_path=tmp_path)
assert fn.is_file()

# we ignore FileNotFoundError
exc_info = (None, FileNotFoundError(), None)
assert not on_rm_rf_error(None, str(fn), exc_info, start_path=tmp_path)

# unknown function
with pytest.warns(pytest.PytestWarning):
exc_info = (None, PermissionError(), None)
Expand Down