S3fs doesn't check again if file exists

[yaelperl1234]

In this code snippet, I check if s3 path exists, then create it by subprocess of aws cli, and then check it again.

import s3fs
import subprocess

s3fs_obj = s3fs.S3FileSystem()

src = 's3://bucket/src'
dst = 's3://bucket/dst'
print(s3fs_obj.exists(dst))
es, out = subprocess.getstatusoutput(f'aws s3 cp {src} {dst} ')

print(es)
print(s3fs_obj.exists(dst))

The results are:

False
0
False

And if I try to open the "dst" path, it fails on "FileNotFoundError".

[martindurant]

This is expected and a necessary optimisation, since listing lookups on s3 are relatively expensive. You can always use invalidate_cache or one of the refresh=True arguments, or don't use the "dircache" at all.

[martindurant]

Arguments controlling the directories cache:

        Parameters
        ----------
        use_listings_cache: bool
            If False, this cache never returns items, but always reports KeyError,
            and setting items has no effect
        listings_expiry_time: int or float (optional)
            Time in seconds that a listing is considered valid. If None,
            listings do not expire.
        max_paths: int (optional)
            The number of most recent listings that are considered valid; 'recent'
            refers to when the entry was set.

[yaelperl1234]

Thank you @martindurant !
But how does it explain the "FileNotFoundError? "open" is a GET request, not list.

[martindurant]

But how does it explain the "FileNotFoundError? "open" is a GET request, not list.

Open needs file information (size, etag, etc), so it also uses the cache. Not sure, but cat/get probably will work fine.