Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(aws): Install AWS CLI v2, upgrade aws-iam-authenticator, remove s3cmd #6156

Merged
merged 3 commits into from Apr 29, 2024
Merged

feat(aws): Install AWS CLI v2, upgrade aws-iam-authenticator, remove s3cmd #6156

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7c781d1
feat(aws): Install AWS CLI v2 and upgrade aws-iam-authenticator
jervi Feb 22, 2024
eb2edc8
Switch to wget (curl isn't available in the slim images)
jervi Feb 22, 2024
6fcdcde
Merge branch 'master' into upgrade_aws_cli
jervi Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 13 additions & 7 deletions Dockerfile.java11.slim
View file
Open in desktop
Expand Up @@ -4,9 +4,8 @@ ARG TARGETARCH

ENV KUBECTL_DEFAULT_RELEASE=1.22.17
ENV KUBECTL_RELEASES="${KUBECTL_DEFAULT_RELEASE} 1.26.12 1.27.9 1.28.5 1.29.0"
ENV AWS_CLI_VERSION=1.22
ENV AWS_CLI_S3_CMD=2.0.2
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.5.9
ENV AWS_CLI_VERSION=2.15.22
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.6.14
ENV GOOGLE_CLOUD_SDK_VERSION=412.0.0
ENV ECR_TOKEN_VERSION=v1.0.2

Expand All @@ -20,11 +19,18 @@ RUN apk update \
wget \
openjdk11 \
git \
openssh-client
openssh-client \
unzip

# AWS CLI
RUN pip install --upgrade --no-build-isolation awscli==${AWS_CLI_VERSION} s3cmd==${AWS_CLI_S3_CMD} python-magic \
&& pip uninstall -y pip
# AWS CLI 2
RUN if [ "${TARGETARCH}" = "arm64" ]; then \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-${AWS_CLI_VERSION}.zip"; \
else \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip"; \
fi && \
unzip awscliv2.zip && \
./aws/install && \
rm -rf ./awscliv2.zip ./aws

# Google cloud SDK
RUN [ $TARGETARCH == 'amd64' ] && export GCP_ARCH="x86_64" || export GCP_ARCH="arm" \
Expand Down
20 changes: 13 additions & 7 deletions Dockerfile.java11.ubuntu
View file
Open in desktop
Expand Up @@ -5,9 +5,8 @@ ENV GOOGLE_CLOUD_SDK_VERSION=412.0.0
ENV PATH "$PATH:/opt/google-cloud-sdk/bin/"
ENV KUBECTL_DEFAULT_RELEASE=1.22.17
ENV KUBECTL_RELEASES="${KUBECTL_DEFAULT_RELEASE} 1.26.12 1.27.9 1.28.5 1.29.0"
ENV AWS_CLI_VERSION=1.22
ENV AWS_CLI_S3_CMD=2.0.2
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.5.9
ENV AWS_CLI_VERSION=2.15.22
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.6.14

RUN apt-get update && apt-get install -y curl gnupg && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
Expand All @@ -20,12 +19,19 @@ RUN apt-get update && apt-get install -y curl gnupg && \
python3-pip \
python3 \
git \
openssh-client && \
openssh-client \
unzip && \
rm -rf ~/.config/gcloud

# AWS CLI
RUN pip install --upgrade --no-build-isolation awscli==${AWS_CLI_VERSION} s3cmd==${AWS_CLI_S3_CMD} python-magic \
&& apt remove -y python3-pip
# AWS CLI 2
RUN if [ "${TARGETARCH}" = "arm64" ]; then \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-${AWS_CLI_VERSION}.zip"; \
else \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip"; \
fi && \
unzip awscliv2.zip && \
./aws/install && \
rm -rf ./awscliv2.zip ./aws

# kubectl + AWS IAM authenticator
RUN for version in $KUBECTL_RELEASES; do \
Expand Down
20 changes: 13 additions & 7 deletions Dockerfile.slim
View file
Open in desktop
Expand Up @@ -4,9 +4,8 @@ ARG TARGETARCH

ENV KUBECTL_DEFAULT_RELEASE=1.22.17
ENV KUBECTL_RELEASES="${KUBECTL_DEFAULT_RELEASE} 1.26.12 1.27.9 1.28.5 1.29.0"
ENV AWS_CLI_VERSION=1.22
ENV AWS_CLI_S3_CMD=2.0.2
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.5.9
ENV AWS_CLI_VERSION=2.15.22
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.6.14
ENV GOOGLE_CLOUD_SDK_VERSION=412.0.0
ENV ECR_TOKEN_VERSION=v1.0.2

Expand All @@ -20,11 +19,18 @@ RUN apk update \
wget \
openjdk17 \
git \
openssh-client
openssh-client \
unzip

# AWS CLI
RUN pip install --upgrade --no-build-isolation awscli==${AWS_CLI_VERSION} s3cmd==${AWS_CLI_S3_CMD} python-magic \
&& pip uninstall -y pip
# AWS CLI 2
RUN if [ "${TARGETARCH}" = "arm64" ]; then \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-${AWS_CLI_VERSION}.zip"; \
else \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip"; \
fi && \
unzip awscliv2.zip && \
./aws/install && \
rm -rf ./awscliv2.zip ./aws

# Google cloud SDK
RUN [ $TARGETARCH == 'amd64' ] && export GCP_ARCH="x86_64" || export GCP_ARCH="arm" \
Expand Down
20 changes: 13 additions & 7 deletions Dockerfile.ubuntu
View file
Open in desktop
Expand Up @@ -5,9 +5,8 @@ ENV GOOGLE_CLOUD_SDK_VERSION=412.0.0
ENV PATH "$PATH:/opt/google-cloud-sdk/bin/"
ENV KUBECTL_DEFAULT_RELEASE=1.22.17
ENV KUBECTL_RELEASES="${KUBECTL_DEFAULT_RELEASE} 1.26.12 1.27.9 1.28.5 1.29.0"
ENV AWS_CLI_VERSION=1.22
ENV AWS_CLI_S3_CMD=2.0.2
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.5.9
ENV AWS_CLI_VERSION=2.15.22
ENV AWS_AIM_AUTHENTICATOR_VERSION=0.6.14

RUN apt-get update && apt-get install -y curl gnupg && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
Expand All @@ -20,12 +19,19 @@ RUN apt-get update && apt-get install -y curl gnupg && \
python3-pip \
python3 \
git \
openssh-client && \
openssh-client \
unzip && \
rm -rf ~/.config/gcloud

# AWS CLI
RUN pip install --upgrade --no-build-isolation awscli==${AWS_CLI_VERSION} s3cmd==${AWS_CLI_S3_CMD} python-magic \
&& apt remove -y python3-pip
# AWS CLI 2
RUN if [ "${TARGETARCH}" = "arm64" ]; then \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-${AWS_CLI_VERSION}.zip"; \
else \
wget -nv -O "awscliv2.zip" "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip"; \
fi && \
unzip awscliv2.zip && \
./aws/install && \
rm -rf ./awscliv2.zip ./aws

# kubectl + AWS IAM authenticator
RUN for version in $KUBECTL_RELEASES; do \
Expand Down
14 changes: 14 additions & 0 deletions clouddriver-web/pkg_scripts/postInstall.sh
View file
Open in desktop
Expand Up @@ -3,6 +3,8 @@
# Remember to also update Dockerfile.*
KUBECTL_DEFAULT_RELEASE=1.22.17
KUBECTL_RELEASES="${KUBECTL_DEFAULT_RELEASE} 1.26.12 1.27.9 1.28.5 1.29.0"
AWS_CLI_VERSION=2.15.22
AWS_AIM_AUTHENTICATOR_VERSION=0.6.14

# ubuntu
# check that owner group exists
Expand All @@ -26,6 +28,18 @@ install_kubectl() {
fi
}

install_awscli2() {
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip" -o "awscliv2.zip"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note arch here is wrong - needs to support arch argument to work on ARM instances.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The postinstall script doesn’t support arm, I just continued down that path

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm, that's... potentially problematic. I need to look at this more then...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can fix it if you want. But I don't think anyone has ever installed the package on an ARM instance. We are probably one of the last companies to actually use these packages, and now we're also migrating to k8s. Looking at the script it is also pretty lacking, without support for anything but AWS. But just let me know if you want me to add support for ARM before we merge this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wince I'd love ARM support, but you're probably right on lack of usage. That said I've missed stuff like this before and it's bitten me :(

unzip awscliv2.zip
./aws/install
rm -rf ./awscliv2.zip ./aws

curl "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${AWS_AIM_AUTHENTICATOR_VERSION}/aws-iam-authenticator_${AWS_AIM_AUTHENTICATOR_VERSION}_linux_amd64" -O aws-iam-authenticator
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arch missing here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

chmod +x ./aws-iam-authenticator \
mv ./aws-iam-authenticator /usr/local/bin/aws-iam-authenticator
}

install_kubectl
install_awscli2

install --mode=755 --owner=spinnaker --group=spinnaker --directory /var/log/spinnaker/clouddriver