Caching fails silently if filepath is too long on windows

[Okabintaro]

I spent a significant amount of time today debugging an issue where a specific function wouldn't get persisted by the cache.

Increasing the verbosity and using the debugger lead me to this Warning in dump_item().

except Exception as e:  # noqa: E722
    warnings.warn(
        "Unable to cache to disk. Possibly a race condition in the "
        f"creation of the directory. Exception: {e}.",
        CacheWarning
    )

The warning didn't make any sense to me since the directory was already created by a previous run.
And the source exception that comes from self.open() -> open() is actually a FileNotFoundError.
In the end an answer on StackOverflow brought me enlightenment.
Turns out that one cached function made the path in the cache hit the Maximum Path Length Limitation on my Windows 10 machine.

Now there is a way to enable support for longer paths but this is not a default on Windows 10 at least. Not sure how it is on Windows 11.

To save future users from the confusion I had I think you could either check for the total length of the absolute path of the cached path or include a hint in that exception warning. At least as long this is still a default behavior on windows.

Maybe one could also try to make the path of the target cache file shorter. It includes the absolute path of the python module which can also be long, depending on its location on the users computer.

[mhegadorn]

Had this same issue today.

Confirmed that older versions would fail silently. However, v1.3.2 at least includes the message No such file or directory without needing to change the verbosity (same Warning as above).

[mhegadorn]

One potential fix could be to specify an extended-length path by adding the prefix \\?\
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation

[viniciusarruda]

I'm using the following to build a path to my Memory instance from joblib.

from joblib import Memory
import os
import pathlib

folder = pathlib.Path(__file__).parent.resolve()

if os.name == "nt":
    _location = os.path.join(folder, "cache")
    if _location.startswith("\\\\"):
        _location = "\\\\?\\UNC\\" + _location[2:]
    else:
        _location = "\\\\?\\" + _location
else:
    _location = os.path.join(folder, "cache")

memory = Memory(location=_location)