Setting up a Kubernetes Cluster with Kops on AWS

Goals

Quickly set up a kube cluster on AWS

Set up an internal registry with authentication

Set up an ingress controller

Prerequisites

Install Docker

Latest instructions should be in Docker’s documentation

Install kubernetes

Latest Instructions should be on kubernetes.io

Install kops

Latest Instructions should be in the Kops README

Install the aws cli

Latest instructions are in the AWS Documentation

Optional, but recommended.

All AWS steps below can be done through the UI, but automating them via the command line will greatly speed up the work.

AWS Account Setup

Create AWS Account if you don’t already have one

Purchase or Transfer Domain, or point DNS to Route53

Generate or upload an SSL Certificate

Preferably covering <APP_DOMAIN>, *.<APP_DOMAIN>, *.kube.<APP_DOMAIN> It needs to cover at least registry.kube.<APP_DOMAIN> and any domains that you will need for your application

We need all of these in a single cert because we will only be using one AWS Load Balancer, because they are expensive. We will use ingress controllers to achieve routing properly.

Log in on the AWS CLI

aws configure

Kops Setup

Set up environment variables

Set the APP_NAME and APP_DOMAIN in environment.sh

Create the S3 bucket for Kops

Kops keeps its cluster configuration information in an s3 bucket, this is also where we will keep the repository of container images for the registry.

source environment.sh # if you have not done it already
aws s3 mb $KOPS_STATE_STORE

Create IAM user for registry

This will be the credentials that the registry uses for storing container images

source environment.sh

mkdir -p tmp
aws iam create-user --user-name registry
aws iam put-user-policy --user-name registry --policy-name kube_bucket_access --policy-document "{ \"Statement\": [ { \"Resource\": [\"arn:aws:s3:::${KUBERNETES_BUCKET_NAME}\",\"arn:aws:s3:::${KUBERNETES_BUCKET_NAME}/*\"],\"Action\": [\"s3:DeleteObject\",\"s3:GetBucketLocation\",\"s3:GetObject\",\"s3:ListBucket\",\"s3:PutObject\",\"s3:PutObjectAcl\"], \"Effect\": \"Allow\" }], \"Version\": \"2012-10-17\" }"
aws iam create-access-key --user-name registry > tmp/registry.json

Update configuration/kube-system/registry-rc.yaml to reflect AWS credentials in tmp/registry.json and to set the bucket path <APP_NAME>_kube

Update configuration/ingress/ingress.kube-system.yaml to reflect the correct domain for incoming requests to the registry

Generate a new SSH key for the kube cluster

source environment.sh
ssh-keygen -t rsa -N "" -f ${APP_KEY_PATH}

Fire up the kops cluster

source environment.sh
kops create cluster $KUBE_CLUSTER_NAME --cloud=aws --dns-zone=$APP_DOMAIN --zones=${AWS_S3_REGION}b,${AWS_S3_REGION}c --node-size=t2.medium --master-size=t2.medium --node-count=4 --ssh-public-key $APP_KEY_PATH.pub --yes

Twiddle Thumbs, consider starting your application container images building while you wait for the cluster to finish starting up.

Common issue may be that the DNS isn’t configured properly, make sure that kube was able to set the dns records for etcd and api. This may take 10-20 minutes

Keep waiting, kube will set up its own DNS records which are required for the next steps

Create registry htpasswd file

This will secure your registry

source environment.sh

mkdir -p configuration/kube-system/registry/
docker run --entrypoint htpasswd registry:2 -Bbn kube <kube_password> >> configuration/kube-system/registry/htpasswd
docker run --entrypoint htpasswd registry:2 -Bbn developer <developer_password> >> configuration/kube-system/registry/htpasswd
kubectl create secret generic registry-auth-secret --from-file=configuration/kube-system/registry/htpasswd --namespace=kube-system

Create image pull secrets from the local registry

source environment.sh

for app_env in {kube-system,default}
do
  kubectl create secret docker-registry internal-registry-login --docker-server registry.kube.${APP_DOMAIN} --docker-username=kube --docker-password=$KUBE_DOCKER_PASSWORD --docker-email=info@rocketmade.com --namespace=$app_env
done

Fire up the registry and ingress services

source environment.sh

kubectl apply -f configuration/kube-system/registry-rc.yaml
kubectl apply -f configuration/kube-system/registry-svc.yaml

for f in configuration/ingress/*.yaml
do
kubectl apply -f $f
done

Point registry.kube.${APP_DOMAIN} to the new ingress load balancer, as an alias

Confirm that you can log in to the registry

source environment.sh
docker login registry.kube.${APP_DOMAIN}

Victory