Getting FileExistsError occasionally

[berkowitze]

I use this function in several different scripts in a project:

@contextmanager
def locked_file(filename: str, mode: str = 'r') -> Generator:
    if mode == 'r' or mode == 'rb' and not os.path.exists(filename):
        raise OSError(f'File {filename} not found.')

    lock_path = filename + '.lock'
    lock = FileLock(lock_path, timeout=10)  # throw error after 10 seconds
    with lock, open(filename, mode) as f:
        try:
            yield f
        finally:
            try:
                os.unlink(lock_path)
            except NameError:
                raise
            except FileNotFoundError:
                pass

# usage example
with locked_file('wow.txt', 'w') as f:
    f.write('hello there')

A little background, this is used on a project that several use on several different machines that share a filesystem. There are also cron jobs running scripts that use locked_file, and subprocesses are sometimes spawned off that use locked_file. Not sure if any of this is relevant, just putting it out there.

Relatively frequently, users get an error that indicates the lock already exists, but it doesn't wait the 10 second timeout seeing if it can acquire the lock it just throws the following exception. Any idea why?

The error goes away upon re-running whatever program originally broke, but it's very frustrating.

In terms of debugging this, I can't reliably reproduce it. I tried writing a script that sleeps for 10 seconds in with block and running it simultaneously from two different machines, but it worked as expected consistently.

Traceback (most recent call last):
  File "/gpfs/main/course/cs1470/admin/course-grading/htabin/../hta/handin/check_submissions.py", line 67, in <module>
    with locked_file(data_file) as f:
  File "/local/projects/python3.7/lib/python3.7/contextlib.py", line 112, in _enter_
    return next(self.gen)
  File "/gpfs/main/course/cs1470/admin/course-grading/hta/handin/helpers.py", line 84, in locked_file
    with lock.acquire(timeout=10), open(filename, mode) as f:
  File "/gpfs/main/course/cs1470/admin/course-grading/ta/venv/lib/python3.7/site-packages/filelock.py", line 251, in acquire
    self._acquire()
  File "/gpfs/main/course/cs1470/admin/course-grading/ta/venv/lib/python3.7/site-packages/filelock.py", line 383, in _acquire
    fd = os.open(self._lock_file, open_mode)
FileExistsError: [Errno 17] File exists: '/course/cs1470/admin/course-grading/ta/assignments.json.lock'

[rogerdahl]

I'm just trying out the filelock module myself. In your function, what happens if a file that would have triggered the check that raises OSError() is created elsewhere between the check and the open()?

[berkowitze]

The function halts after the raise, so wherever that problematic locked_file was called will fail as well.
If the file is removed between the check and the open, that's more problematic you'll get a hanging lock file and not sure if it gets released. In my application, files aren't really removed so I'm not concerned about this personally.

[Zooce]

Looking at the code, in UnixFileLock._acquire the call to os.open is not wrapped in a try-except, but WindowsFileLock._acquire does wrap the os.open call in a try-except.

[gaborbernat]

Hello, if you make a PR for this (with tests) we would be happy to review it, thanks!