Preload images of aws-node, kube-proxy and possible others

[runningman84]

What I'd like:
I would like to have some images included in the bottlerocket amis like

aws node

602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni-init:v1.16.3-eksbuild.2
602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon/aws-network-policy-agent:v1.0.8-eksbuild.1 (this takes about 30s to download)

aws guard duty

409493279830.dkr.ecr.eu-central-1.amazonaws.com/aws-guardduty-agent:v1.4.1

aws ebs

602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.28.0
602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-node-driver-registrar:v2.10.0-eks-1-29-5
602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.12.0-eks-1-29-5

kubeproxy

602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/kube-proxy:v1.29.1-minimal-eksbuild.2

From my point of view it should these managed extension by AWS should be included in the latest and previous version for the current k8s release.

This would really speedup node start time...

Any alternatives you've considered:

[yeazelm]

Hello @runningman84, thanks for cutting this issue! I believe this is a similar request as #614 and #2023 where you would like images cached/preloaded before boot. The list you provided is a good list but I don't think every use cases needs all of these images. I think the better approach might be to provide a mechanism to do this caching for the containers you care about.

[runningman84]

I think in aws everything but the guardduty stuff is default in any cluster. Even if one or two images are not needed that would not hurt because unused images would be cleaned up if the disk is too full.

[misterek]

This would be a way you could do it yourself: https://github.com/aws-samples/bottlerocket-images-cache/blob/main/README.md

[runningman84]

Yes that is possible but I would rather like a default solution without much complexity. Because AWS is releasing bottlerocket and other updates in a regular interval it would make sense to have it included out of the box.

You could even argue that this would help in the marketing because AWS might be the fastest provider to spin up new kubernetes nodes serving customer workloads...

[sanjinp]

This would be a way you could do it yourself: https://github.com/aws-samples/bottlerocket-images-cache/blob/main/README.md

While this might be viable option for smaller images, it showed to be suboptimal for caching larger ones (1GB+) as container cold start improvements would be masked due to underlying stack of how snapshots are working - backed by s3.
So you will see your Pod in running state quite fast in few seconds, but our tests showed that our cold-start performance dropped by a half with snapshot pre-caching opposed to fresh download of an image.
Once you start Pod it needs to access certain volume blocks from snapshot (saved on s3), and from that point on it starts downloading them from s3 (warming that part of volume).
So result for our 4GB image:

• direct download from ECR ~3 mins to start processing
• with baked images into snapshot ~7+ minutes to start processing
  Although there is EBS FSR available, it is too expensive for such cases.