Unauthorized tagging operation

[bergmorten]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v20.12.2

Amplify CLI Version

12.11.1

What operating system are you using?

windows

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

nope

Describe the bug

I attempted to add a custom resource that enables a backup plan with a KMS key (based on https://aws.amazon.com/blogs/storage/automate-backups-for-aws-amplify-graphql-backends-with-aws-backup/).

I've updated the CDK and changed some names and the policy owner in the example code, which I do not believe is the issue (see attached CDK code if needed).

The error I receive does not seem relevant and links to an access policy that I do not process when I execute amplify push.

🛑 CFN Deployment failed for custom resources.
Name: amplifyclientdevelopkeyA93F5F9B (AWS::KMS::Key), Event Type: create, Reason: Resource handler returned message: "Encountered a permissions error performing a tagging operation, please add required tag permissions. See https://repost.aws/knowledge-center/cloudformation-tagging-permission-error for how to resolve. Resource handler returned message: "Unauthorized tagging operation"" (RequestToken: e3913e9e-a5bf-ef1a-befa-81f271edd41e, HandlerErrorCode: UnauthorizedTaggingOperation), IsCustomResource: true

Is this an Amplify error, a cdk-script error or do I need to fellow the steps in the link?

cdk-stack.ts

import * as AmplifyHelpers from '@aws-amplify/cli-extensibility-helper';
import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as kms from 'aws-cdk-lib/aws-kms';
import * as backup from 'aws-cdk-lib/aws-backup';
export class cdkStack extends cdk.Stack {
  constructor(
    scope: Construct,
    id: string,
    props?: cdk.StackProps,
    amplifyResourceProps?: AmplifyHelpers.AmplifyResourceProps,
  ) {
    super(scope, id, props);
    /* Do not remove - Amplify CLI automatically injects the current deployment environment in this input parameter */
    new cdk.CfnParameter(this, 'env', {
      type: 'String',
      description: 'Current Amplify CLI env name',
    });

    // This line is changed
    const keyAdmin = new iam.AccountRootPrincipal();
    // This line is changed
    const backupAdmin = new iam.Role(this, 'BackupAdminRole', {
      assumedBy: new iam.AccountRootPrincipal(),
    });
    backupAdmin.addManagedPolicy(
      iam.ManagedPolicy.fromAwsManagedPolicyName('AWSBackupFullAccess'),
    );

    const key = new kms.Key(
      this,
      `amplify-client-${AmplifyHelpers.getProjectInfo().envName}-key`,
      {
        removalPolicy: cdk.RemovalPolicy.RETAIN,
        alias: `alias/amplify-client-${
          AmplifyHelpers.getProjectInfo().envName
        }-key`,
        description:
          'KMS key for encrypting the objects in your AWS Backup Vault',
        enableKeyRotation: false,
        admins: [backupAdmin, keyAdmin],
        policy: new iam.PolicyDocument({
          statements: [
            new iam.PolicyStatement({
              actions: [
                'kms:Create*',
                'kms:Describe*',
                'kms:Enable*',
                'kms:List*',
                'kms:Put*',
                'kms:Update*',
                'kms:Revoke*',
                'kms:Disable*',
                'kms:Get*',
                'kms:Delete*',
                'kms:ImportKeyMaterial',
                'kms:TagResource',
                'kms:UntagResource',
                'kms:ScheduleKeyDeletion',
                'kms:CancelKeyDeletion',
              ],
              principals: [keyAdmin],
              resources: ['*'],
            }),
            new iam.PolicyStatement({
              actions: [
                'kms:Encrypt',
                'kms:Decrypt',
                'kms:ReEncrypt*',
                'kms:GenerateDataKey*',
                'kms:DescribeKey',
              ],
              principals: [backupAdmin],
              resources: ['*'],
            }),
          ],
        }),
      },
    );

    const plan = new backup.BackupPlan(this, 'amplify-client-plan', {
      backupPlanName: `amplify-client-plan-${cdk.Fn.ref('env')}`,
      backupPlanRules: [
        backup.BackupPlanRule.daily(),
        backup.BackupPlanRule.weekly(),
        backup.BackupPlanRule.monthly1Year(),
      ],
      backupVault: new backup.BackupVault(this, 'Vault', {
        backupVaultName: `amplify-client-vault${
          AmplifyHelpers.getProjectInfo().envName
        }`,
        encryptionKey: key,
        accessPolicy: new iam.PolicyDocument({
          statements: [
            new iam.PolicyStatement({
              effect: iam.Effect.DENY,
              principals: [new iam.AnyPrincipal()],
              actions: ['backup:DeleteRecoveryPoint'],
              resources: ['*'],
              conditions: {
                StringNotLike: {
                  'aws:PrincipalArn': [backupAdmin.roleArn],
                },
              },
            }),
          ],
        }),
      }),
    });

    plan.addSelection(`ClientSelectionByTag`, {
      resources: [
        backup.BackupResource.fromTag('user:Stack', cdk.Fn.ref('env')),
      ],
    });
  }
}

Expected behavior

Should not complain about tagging permissions?

Reproduction steps

1. amplify add custom
2. replace cdk-stack.ts with the attached code above
3. amplify push

Project Identifier

Project Identifier: 0e0c35de10e80e85c555ebedce4575e5

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey @bergmorten, thank you for reaching. From the error message it appears the profile being used project does not have tagging permission for the KMS resource on a custom resource.
Could you try adding the tagging permission as an inline policy on the IAM user profile then run push?
https://docs.aws.amazon.com/kms/latest/APIReference/API_TagResource.html

[bergmorten]

I just verified my IAM profile and I'm member of admin group, and the policy of AdministratorAccess and does indeed include TagResource in KMS. Could it be that amplify use another profile?

I've successfulle creared other custom resource before but not KMS and and with latest amplify CLI.

[ykethan]

Hey @bergmorten, we can check the profile the project is using in the .config/local-aws-info.json file. The .config folder may be hidden in Vscode, you can disable this in the .vscode/settings.json.
On testing with the provided example, i did run into the error with a user that utilized AdministratorAccess-Amplify. On adding a inline permission with tagging permissions and other resource permissions such as backup,kms I was able to do a successful push.

[bergmorten]

Thanks I'll try this :-)

[ykethan]

@bergmorten do reach out to us if you are experiencing issues with this.

[bergmorten]

@ykethan An delayed update ( took a few day off last week).

First I tried to add inline policy for my IAM account:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"kms:TagResource",
				"kms:UntagResource"
			],
			"Resource": "*"
		}
	]
}

This did not change anything. Then I tried to deploy via amplify console / continuous deploys trigger. This also resulted in the same error. Amplify console use the amplifyconsole-backend-role which has full AdministratorAccess.

I need continuous deployment to work since I've multiple stacks/tenants which triggers deployment. I do not understand why AdministrorAccess is not sufficient?

[ykethan]

@bergmorten retested this, but did not observe this behavior on my push.
user permissions

created a custom resource with the code provided and pushed

modify the permissions to your requirements enabling only the permissions needed for the operation

[bergmorten]

Very odd, I see that you have kms:* , while I only set kms:TagResource and kms:UnTagResource. Must I do re-login for the policy to be effective?

Still I think it very odd that administrator access (you also has that) which has Action:* is not sufficient? Why is inline policy better than AWS managed roles?

[ykethan]

Hey @bergmorten, with AWS Admin access should have access to creating the resources. In my testing i used a user with Amplify Admin access which is a scoped policy for Amplify resources. I utilized the inline permission to provide additional permission which in your case should not be needed as it is AWS Admin access.
As a quick check could you ensure the access key matches the .config/local-aws-info.json profile credentials locally present at ~/.aws/credentials for example.

[bergmorten]

I got the aws-amplify-console to push the custom CDK, the error I had previously with console was not a tag resource error 👍 Since I now have pushed the CDK, it think it will be more difficult to debug the CLI access issue.

My content of amplify/.config/local-aws-info.json is:

{
  "develop": {
    "configLevel": "amplifyAdmin"
  }
}

Since it is missing profileName and useProfile, I assume it use 'default'?

In ~/.aws/credentials I've two accounts, default and amplify-account, both with an aws_access key & secret (no SSO).

The account amplify-account point to a IAM user which has only one policy attached (administratorAccess).

However the default points to a IAM user with multiple policies, including both administrator and administratorAccess-Amplify.

The default administrator has type job function, could this be an issue?
or, is it a issue that default has multiple policies where some allow other not? I thought this was ok...

[ykethan]

Hey @bergmorten, could you try running amplify configure project -> AWS Profile setting -> yes -> Select the authentication method you want to use: AWS profile and select the profile. This update the project to use the profile. Could you let us know if this mitigates the push issue. Do ensure you run a pull first to sync the changes made on the console build.

[bergmorten]

@ykethan I do confirm that this did work. I did checkout of an another environment, which did not yet had custom backup pushed. Then updated my amplify -> aws profile setting, and after that a amplify pull & push worked :-)

Thanks for your help.

I think you should consider to make backup and a restore guide a part of the amplify tools and guides. It's a bit odd that a such important task is not part of amplify?

[ykethan]

Hey @bergmorten, we recently introduced Amplify Gen 2, which provides advanced customizations and integrations using CDK under the hood.
Do check out our documentation at Amplify documentation and let us know if this would assist with your use case. Examples on deletion protection and backup: https://docs.amplify.aws/react/build-a-backend/add-aws-services/deletion-backup-resources/

[juanf9224]

Hi @bergmorten @ykethan , chiming in as I was having this issue as well trying to create a KMS Key through Amplify custom resource with CDK, been stalking this GH issue for clues 🤣 , and finally what worked for me was to update the .config/local-aws-info.json so that it uses the default profile instead of the Amplify-Admin one, not sure why it would fail with the amplify one as it has the TagResource and UntagResource permission on CloudFormation actions, only thing I see is that it has a string like pattern check on the resources property, could this be affecting the create process in this case?: