For users missing certain S3 permissions, files may be identified as directories

[adrianohrl]

Depending on the machine in which the upload happens, downloading works fine. However, when a different machine tries to download the S3 remote path, it downloads files as directory.

from pathlib import Path

from cloudpathlib import CloudPath

CWD = Path.cwd()
TMP = Path(tempfile.gettempdir())

remote_path = CloudPath(f's3://test/models/')

local_path = CWD / 'models/'
remote_path.upload_from(local_path)

for rp in remote_path.iterdir():
    lp = local_path / rp.relative_to(remote_path)
    assert rp.is_file() == lp.is_file(), f'The {rp} remote path must be a file!'
    assert rp.is_dir() == lp.is_dir(), f'The {rp} remote path must be a directory!'

local_path = TMP / 'models'
remote_path.download_to(local_path)

for rp in remote_path.iterdir():
    lp = local_path / rp.relative_to(remote_path)
    assert rp.is_file() == lp.is_file(), f'The {lp} local path must be a file!'
    assert rp.is_dir() == lp.is_dir(), f'The {lp} local path must be a directory!'

[adrianohrl]

Actually, I found the issue was related to the IAM policy. My local role is different of the remote one. I am not sure yet what is the solution.

Shouldn't cloudpathlib raise an Exception in this case?

[adrianohrl]

In the cloudpathlib.s3.s3client.py:_s3_file_query, raised a ClientError due to HeadObject forbiddance. Then, on except, it considers it as a prefix.

[pjbull]

@adrianohrl Sounds like the IAM had list permissions, but not some other permission (like GetObject or GetObjectAttributes). Do you know which ones you missing? That will be helpful.

We'll need to double check, but I think you may get a 403 both in the case that you do no have permissions for head_object and in the case that the object does not exist, which may complicate a fix here. If so, we may consider doing both a list check and then a get check on instantiation to ensure the user has the requisite permissions.