Can't Delete Objects?

[mexicantexan]

This has been beaten to death, but I am genuinely confused. I am trying to use boto3 to delete objects and all of their versions. Here is my script:

def delete_file_boto3(pathList, s3_resource=None, parent_path=None):
    if len(pathList) < 1:
        return True, [], ''
    path = pathList[0]
    errors = []
    parent_path = normalize_address(parent_path)
    client = boto3.client('s3', aws_access_key_id=AWS_ACCESS_KEY_ID,
                          aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
    paginator = client.get_paginator('list_object_versions')
    if parent_path:
        response_iterator = paginator.paginate(Bucket=AWS_STORAGE_BUCKET_NAME, Prefix=parent_path)
    else:
        response_iterator = paginator.paginate(Bucket=AWS_STORAGE_BUCKET_NAME)
    for response in response_iterator:
        versions = response.get('Versions', [])
        versions.extend(response.get('DeleteMarkers', []))
        try:
            for path in pathList:
                for x in response['Versions']:
                    try:
                        if x['Key'] == path:
                            print('Deleting', x['Key'], x['VersionId'])
                            client.delete_object(Bucket=AWS_STORAGE_BUCKET_NAME, Key=x['Key'],
                                                 VersionId=x['VersionId'])
                    except KeyError:
                        print('key error on:', response)
                        errors.append(path)
                        continue
                    except botocore.exceptions.ClientError:
                        print(traceback.format_exc())
                try:
                    if x['Key'] == path:
                        # sanity check
                        client.delete_object(Bucket=AWS_STORAGE_BUCKET_NAME, Key=x['Key'])
                except:
                    # expected behavior
                    pass
        except KeyError:
            print('key error on:', response)
            errors.append(path)

    if len(errors) == 0:
        success = True
    else:
        success = False
    return success, errors, path

For context, I can successfully upload into the bucket. I have tried setting AWS_DEFAULT_ACL to 'public-read' and None. The bucket does not block public access. The account I am using has AmazonS3FullAccess. Object Ownership in the bucket is set to ACLs enabled. I don't have MFA for deleting objects turned on for deleting versioned files. Still I get the following error: botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied

Any suggestions?

[RyanFitzSimmonsAK]

Hi @mexicantexan, thanks for reaching out. Could you please attach debug logs of a run of your script? You can get debug logs by adding boto3.set_stream_logger('') to your script, and redacting any sensitive information. Thanks!

[mexicantexan]

versionId=UWx5u2FHsTdBeKoTV5Zbj6pWFeuQoBvj
host:{{REDACTED hostname; what printed was correct}}.s3.us-west-2.amazonaws.com
x-amz-content-sha256:{{REDACTED 64 char string}}
x-amz-date:20230208T001149Z

host;x-amz-content-sha256;x-amz-date
{{REDACTED 64 char string}}
2023-02-08 00:11:49,929 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20230208T001149Z
20230208/us-west-2/s3/aws4_request
{{REDACTED 64 char string}}
2023-02-08 00:11:49,929 botocore.auth [DEBUG] Signature: {{REDACTED 64 char string}}
2023-02-08 00:11:49,930 botocore.hooks [DEBUG] Event request-created.s3.DeleteObject: calling handler <function add_retry_headers at 0x7fe83b0ca700>
2023-02-08 00:11:49,930 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://{{REDACTED hostname; what was printed was correct}}.s3.us-west-2.amazonaws.com/1xv4ikuo1w/products/9w2keoc5jf/Miscellaneous/original/generate_trimaps.py?versionId=UWx5u2FHsTdBeKoTV5Zbj6pWFeuQoBvj, headers={'User-Agent': b'Boto3/1.26.64 Python/3.8.16 Linux/5.15.0-58-generic Botocore/1.29.64', 'X-Amz-Date': b'20230208T001149Z', 'X-Amz-Content-SHA256': b'{{REDACTED 64 char string}}', 'Authorization': b'AWS4-HMAC-SHA256 Credential={{REDACTED}}/20230208/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature={{REDACTED 64 char string}}', 'amz-sdk-invocation-id': b'ea80550b-5b68-4674-ae9f-2b2717254df7', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2023-02-08 00:11:49,931 botocore.httpsession [DEBUG] Certificate path: {{REDACTED}}/venv/lib/python3.8/site-packages/certifi/cacert.pem
2023-02-08 00:11:49,997 urllib3.connectionpool [DEBUG] https://{{REDACTED hostname; what was printed was correct}}.s3.us-west-2.amazonaws.com:443 "DELETE /1xv4ikuo1w/products/9w2keoc5jf/Miscellaneous/original/generate_trimaps.py?versionId=UWx5u2FHsTdBeKoTV5Zbj6pWFeuQoBvj HTTP/1.1" 403 None
2023-02-08 00:11:49,998 botocore.parsers [DEBUG] Response headers: {'x-amz-request-id': '00GJ98SCVH3T031H', 'x-amz-id-2': '{{REDACTED Host ID 75 characters long}}', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Wed, 08 Feb 2023 00:11:49 GMT', 'Server': 'AmazonS3'}
2023-02-08 00:11:49,998 botocore.parsers [DEBUG] Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>00GJ98SCVH3T031H</RequestId><HostId>{{REDACTED Host ID 75 characters long}}</HostId></Error>'
2023-02-08 00:11:49,999 botocore.hooks [DEBUG] Event needs-retry.s3.DeleteObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fe829d0c730>
2023-02-08 00:11:49,999 botocore.retryhandler [DEBUG] No retry needed.
2023-02-08 00:11:49,999 botocore.hooks [DEBUG] Event needs-retry.s3.DeleteObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fe832743040>>
trying different version
Traceback (most recent call last):
  File "{{REDACTED}}/file_name_functions.py", line 656, in delete_file_boto3
    client.delete_object(Bucket=AWS_STORAGE_BUCKET_NAME, Key=x['Key'],
  File "{{REDACTED}}/venv/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "{{REDACTED}}/venv/lib/python3.8/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied

[mexicantexan]

@RyanFitzSimmonsAK Thanks for taking this up, I really do appreciate it. I think I redacted all the sensitive info. If I redacted too much, I can always add it back in for you.

[mexicantexan]

I'm wondering if it has to do with my bucket policy, this is the pertinent bucket policy for the user I'm using for boto3 access:

{
    "Version": "2012-10-17",
    "Id": "Policy1234567890987",
    "Statement": 
        [...,
          {
              "Sid": "API Access",
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::{{REDACTED}}:user/{{REDACTED_USER_NAME}}"
              },
              "Action": "s3:*",
              "Resource": [
                  "arn:aws:s3:::{{REDACTED_BUCKET_NAME}}/*",
                  "arn:aws:s3:::{{REDACTED_BUCKET_NAME}}"
              ]
         }
      ]
}

I feel like it shouldn't though because I think that above policy should let boto3 do whatever it wants, right?

[mexicantexan]

For more context, the user account that I'm using is not the bucket owner either which is why I appended the above bucket policy.

[RyanFitzSimmonsAK]

Can you double check that the account you're using for boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) is the same that you've given permissions to? Your policy looks correct, so it could be an issue with your credentials. Thanks!

[mexicantexan]

I went ahead and hard-coded the Access Key and it's Secret into the app. I already had the environment variables AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID set in the application, so I installed the aws cli and ran the aws configure to set the options on my dev machine for good measure. Restarted the computer just for even more ?good measure? 😅 and no dice. :/ I can provide screenshots to verify this if needed.

[mexicantexan]

I feel dumb, looking closer at the target user, there was a policy that I had not attached to the user called AWSCompromisedKeyQuarantineV2, I think this was automatically applied?? After setting up a new key, everything is working properly. Thanks @RyanFitzSimmonsAK for your patience and help on all of this!