Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(aws_ecs): CDK insists on Modifying the ECS Execution/Task Role with duplicate permissions, resulting policy Is too big to deploy #18926

Closed
mdesouky opened this issue Feb 10, 2022 · 4 comments
Closed

(aws_ecs): CDK insists on Modifying the ECS Execution/Task Role with duplicate permissions, resulting policy Is too big to deploy #18926

mdesouky opened this issue Feb 10, 2022 · 4 comments
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. p1

Comments

@mdesouky
Copy link

mdesouky commented Feb 10, 2022 •
edited

What is the problem?

Trying to create a scheduled fargate task, I created the necessary IAM role to use as both Execution Role and Task Role in the definition as follows:
IAM Role:

def ecs_execution_role(self, role_name: str, props):
    ecs_execution_role = iam.Role(
        scope=self,
        id='CoreECSExecutionRole',
        role_name=role_name,
        managed_policies=[iam.ManagedPolicy.from_managed_policy_arn(
            scope=self,
            id='AmazonECSTaskExecutionRolePolicy',
            managed_policy_arn="arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
        )],
        assumed_by=iam.CompositePrincipal(
            iam.ServicePrincipal("ecs.amazonaws.com"),
            iam.ServicePrincipal("ecs-tasks.amazonaws.com")
        )
    )
    ecs_execution_role_default_policy = iam.Policy(
        scope=self,
        id="ECSExecutionRoleDefaultPolicy",
        statements=[
            iam.PolicyStatement(
                sid='GetSecrets',
                effect=iam.Effect('ALLOW'),
                actions=[
                    "secretsmanager:GetSecretValue",
                    "secretsmanager:DescribeSecret"
                ],
                resources=[f"arn:aws:secretsmanager:{props['region']}:{props['account_id']}:secret:/{props['org']}/{props['component']}/{props['environment']}/*"]
            ),
        ]
    )
    ecs_execution_role.attach_inline_policy(ecs_execution_role_default_policy)
    return ecs_execution_role

Passing it to the function creating the task:

#Create ECS Role
            core_ecs_role = ecs_execution_role(self, 'core-ecs-role', props)
            #Create ECS Task
            create_advertising_task(self, 'AdvertisingCronTask', fargate_cluster, core_ecs_role, props, core_secrets, core_loggroup)

The Code Creating the task:

def create_advertising_task(self, name: str, ecs_cluster: ecs.ICluster, role: iam.IRole, props: map, core_secrets: map, loggroup: logs.ILogGroup):
    #create ecs_secrets out of core_secrets
    #since Object of type aws-cdk-lib.aws_secretsmanager.Secret is not convertible to aws-cdk-lib.aws_ecs.Secret
    ecs_secrets = {}
    for key, value in core_secrets.items():
        ecs_secrets[key] = ecs.Secret.from_secrets_manager(value)
    advertising_task_definition = ecs.FargateTaskDefinition(
        scope=self,
        id=f"{name}Definition",
        memory_limit_mib=props['advertising']['memory'],
        execution_role=role,
        task_role=role
    )
    advertising_repo = ecr.Repository.from_repository_arn(self, "CoreApiAdvertising", "arn:aws:ecr:ap-southeast-2:444436158009:repository/core-api-advertising")
    advertising_task_definition.add_container(
        'AdvertisingContainer',
        container_name=f"{props['component']}-advertising-cron-{props['environment']}",
        image=ecs.EcrImage(advertising_repo, props['advertising']['image_tag']),
        command=['python', 'manage.py', 'advertising_generate_ad_queues', '1'],cpu=props['advertising']['cpu'],
        memory_limit_mib=props['advertising']['memory'],
        secrets=ecs_secrets,
        logging=ecs.LogDriver.aws_logs(
            stream_prefix="ecs/fargate",
            log_group=loggroup
        )

    )
    advertising_task = ecs_patterns.ScheduledFargateTask(
        scope=self,
        id=name,
        scheduled_fargate_task_definition_options=ecs_patterns.ScheduledFargateTaskDefinitionOptions(
            task_definition=advertising_task_definition
        ),
        schedule=applicationautoscaling.Schedule.expression(f"cron({props['advertising']['cron']})"),
        cluster=ecs_cluster,
        enabled=props['advertising']['enabled'],
        rule_name='AdvertisingCronTrigger',
        platform_version=ecs.FargatePlatformVersion.LATEST

    )

The resulting CFN template has got extra permissions added to the role created by myself and not the task definition construct, wouldn't be such a big problem if it weren't for the huge number of secrets we're using which made the policy too big to deploy
got the follwowing error from CFN

Maximum policy size of 10240 bytes exceeded for role qsic-core-cdk-stack-05050-AdvertisingCronTaskDefin-1K9NOKVNF47VG (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded; Request ID: ebf95d73-6cdc-415e-89a5-edd1c25eec05; Proxy: null)

here is the resulting CFN template

CoreECSExecutionRole9E22056A:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs.amazonaws.com
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      RoleName: core-ecs-role
      Tags:
        - Key: Branch
          Value: None
        - Key: Component
          Value: core
        - Key: CostCenter
          Value: Cloud Management
        - Key: DataClassification
          Value: Internal
        - Key: Environment
          Value: dev
        - Key: Repo
          Value: https://bitbucket.org/qsic/qsic-service-catalog/src/master/
    Metadata:
      aws:cdk:path: qsic-core-cdk-stack-050507561652-dev/CoreECSExecutionRole/Resource
  CoreECSExecutionRoleDefaultPolicyC08420B6:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - ecr:BatchCheckLayerAvailability
              - ecr:GetDownloadUrlForLayer
              - ecr:BatchGetImage
            Effect: Allow
            Resource: arn:aws:ecr:ap-southeast-2:444436158009:repository/core-api-advertising
          - Action: ecr:GetAuthorizationToken
            Effect: Allow
            Resource: "*"
          - Action:
              - logs:CreateLogStream
              - logs:PutLogEvents
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - AdvertisingCronTaskDefinitionAdvertisingContainerLogGroup6FD2BF8C
                - Arn
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ENVIRONMENTF9918D0A
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: NEWRELICAPPLICATIONIDFC394F28
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: PUBNUBSUBSCRIBEF16CB97A
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITDBHOST93BE01B7
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: REMOT3BASEURL35831472
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: REMOT3USERNAME0C819A27
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SENTRYDSN535DD920
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SLACKWEBHOOK819671D5
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQLDBUSER4A33352E
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: STATICROOT33176076
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ADBRIEFEMAIL1B42E448
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ESAPMSECRETTOKEN6371CE0F
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ESAPMSERVICENAME7E5C0B08
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: FIREBASEURL7ECBDFE2
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITDBPASSWORD841347BB
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITAPIURLAA628C55
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: S3CONTENTBUCKETBD28C8CC
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SLACKURL2282D6BF
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQLDBHOSTCEEE3C24
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQSQUEUEADTRANSCODEB35E4641
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ADDITIONALALLOWEDHOSTS4CB53727
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: BASEHQADMINURIB6F37705
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: DJANGOLOGLEVELADCAD51D
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: MAILGUNAPIKEY51730C22
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: NOREPLYEMAIL7FBBEFAD
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: PUBNUBSECRET71518FA2
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: REMOT3PASSWORD80B9F6DF
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SONOSAPIAUTHTOKEN6100545B
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQLDBNAMEB9087282
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: TRACKTHUMBNAILURLCEDA35EC
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: AWSSTORAGEBUCKETNAMECFA74999
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: BASEHQURI66205524
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ESCLOUDAUTH317A1AD7
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: NEWRELICAPIKEY1E785D5E
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: NEWRELICENV55C5CFDA
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: PUBNUBPUBLISHE7549E0C
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SENTRYWEBHOOKDFC90EDB
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SLACKUSERNAMEDAD090C1
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SONOSGETTOKEN0493DEB3
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQLDBPASSWORD4CAFCAC5
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: AWSLOCATION3B2C31CE
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ELASTICSEARCHURL63D4FC95
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ESAPMSERVICEURL44937253
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITDBNAME85F7FA9E
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITDBPORT8FDCF066
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITDBUSER710CD655
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: RAVENDSNE674CEDE
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SEAPIKEY02EE70B6
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SLACKCHANNEL70F2C590
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SQLDBPORTCC2FBFE5
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: AWSREGION12990A67
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: BASEMEDIAURL2622A1B6
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: ESCLOUDIDC4867181
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: LOGSTASHURL75181291
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: MAILGUNBASEURI74D4B084
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: QBITAPITOKEN62058CBB
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: REMOT3APIKEYEAA83F55
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SECRETKEYC195FE10
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource:
              Ref: SONOSCREATEUSER3DE67375
        Version: "2012-10-17"
      PolicyName: CoreECSExecutionRoleDefaultPolicyC08420B6
      Roles:
        - Ref: CoreECSExecutionRole9E22056A
    Metadata:
      aws:cdk:path: qsic-core-cdk-stack-050507561652-dev/CoreECSExecutionRole/DefaultPolicy/Resource
  ECSExecutionRoleDefaultPolicyE7726C43:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
            Effect: Allow
            Resource: arn:aws:secretsmanager:ap-southeast-2:050507561652:secret:/qsic/core/dev/*
            Sid: GetSecrets
        Version: "2012-10-17"
      PolicyName: ECSExecutionRoleDefaultPolicyE7726C43
      Roles:
        - Ref: CoreECSExecutionRole9E22056A
    Metadata:
      aws:cdk:path: qsic-core-cdk-stack-050507561652-dev/ECSExecutionRoleDefaultPolicy/Resource

Reproduction Steps

cdk synth

What did you expect to happen?

only permissions created to the role would be there

What actually happened?

CDK added duplicate permission to the role not created by the the task definition construct

CDK CLI Version

2.12.0 (build c9786db)

Framework Version

No response

Node.js Version

v17.0.1

OS

MacOs Monterey,

Language

Python

Language Version

Python 3.8.8

Other information

Get the same results running in a docker container based on python:3.8-slim
@mdesouky mdesouky added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 10, 2022
@github-actions github-actions bot added the @aws-cdk/aws-ecs Related to Amazon Elastic Container label Feb 10, 2022
@NGL321 NGL321 added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Feb 14, 2022
@peterwoodworth
Copy link
Contributor

Thanks for reporting this @mdesouky,

We're aware of this issue, and haven't had the time to put in the fix for it since messing with policies is really tricky, but is a priority to get done this year. We're tracking the issue in a few places, as there are a couple different ways we can help to prevent this error. Here are a few places where we're tracking it: #16244 #14713 and a PR #16350

Please give thumbs up on these issues, that is the best way to help us prioritize them 🙂

Ping me if you have any other concerns

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@peterwoodworth
Copy link
Contributor

The additional policies are being added because when you call add_container and pass in the secrets, the CDK will grant the execution role read permissions for each secret

aws-cdk/packages/@aws-cdk/aws-ecs/lib/container-definition.ts

Lines 455 to 467 in 22b034f

if (props.secrets) {
this.secrets = [];
for (const [name, secret] of Object.entries(props.secrets)) {
if (secret.hasField) {
this.referencesSecretJsonField = true;
}
secret.grantRead(this.taskDefinition.obtainExecutionRole());
this.secrets.push({
name,
valueFrom: secret.arn,
});
}
}

The CDK will create a new policy as a child of the execution role you've created. You can access and override this policy with escape hatches

cfn_policy = ecs_execution_role.node.find_child('DefaultPolicy');
cfn_policy.addPropertyOverride('PolicyDocument', policy_document)

@NGL321 NGL321 mentioned this issue Mar 8, 2022
Closed
@bearrito
Copy link

I'm seeing a similar issue but with logging. How does this escape hatch mechanism work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. p1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bearrito @NGL321 @mdesouky @peterwoodworth @madeline-k