Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

please ditch fcache as it does not work with multiprocessing on Windows #303

Open
masariello opened this issue Mar 6, 2024 · 3 comments
Assignees

Comments

@masariello
Copy link

Describe the bug

fcache has a known issue with multiprocessing (tsroten/fcache#26) that has been ignored tor 5 years.

Here's an example of the errors

2024-03-06 10:54:33.478 - ERROR - apscheduler.executors.unleash_executor_YTIN7P - run_job - Job "fetch_and_load_features (trigger: interval[0:01:00], next run at: 2024-03-06 10:55:32 CST)" raised an exception
Traceback (most recent call last):
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\apscheduler\executors\base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "C:\Users\dchuserNameoi\AppData\Roaming\Python\Python39\site-packages\UnleashClient\periodic_tasks\fetch_and_load.py", line 43, in fetch_and_load_features
    cache.set(ETAG, etag)
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\UnleashClient\cache.py", line 129, in set
    self._cache[key] = value
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\fcache\cache.py", line 267, in __setitem__
    self._write_to_file(filename, value)
  File "C:\Users\userName\AppData\Roaming\Python\Python39\site-packages\fcache\cache.py", line 254, in _write_to_file
    os.chmod(filename, self._mode)
OSError: [WinError 6800] The function attempted to use a name that is reserved for use by another transaction: 'C:\\Users\\userName\\AppData\\Local\\PROD\\PROD\\Cache\\cache\\65746167'

To Reproduce

Please see StackOverflow linked in tsroten/fcache#26

@sighphyre sighphyre self-assigned this Mar 13, 2024
@sighphyre
Copy link
Member

Hey @masariello, this probably isn't something that'll get resolved in the next few weeks. But we do have tentative plans to rework the caching anyway due to an upcoming project on this SDK

Is this actually affecting you, outside of causing trouble in the logs? Writing to backup, shouldn't be a process that needs to happen to ensure the SDK works correctly

@masariello
Copy link
Author

masariello commented Mar 13, 2024

Yes. I hit the problem reproduced in the linked fcache issues

In fact, fcache has numerous issues when running with multiprocessing. It just opens and writes files on disk without any locking or retry logic, so every now and then we do get exceptions caused by those concurrent accesses.

The only way to avoid those issues is replacing fcache with a dict-based BaseCache

@calum-pledge-io
Copy link

As an extra motivation, fcache also isn't compatible with many serverless environments out-of-the-box.
For example, in AWS lambda, the home directory is mounted on a read-only file system and fcache defaults to $HOME/.cache/ as the base path for cache files. This can be worked around by setting XDG_CACHE_HOME to something like /tmp/cache/, but it makes this package slightly more cumbersome to use in a serverless deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

3 participants