Add kind-based development environment #422

nckturner · 2022-01-18T02:25:53Z

Scripts to create and tear down the environment.
Uses a kind cluster and a stand-alone authenticator container.
Sets up a kubeconfig to test client-server interactions end to end.

Example use:

$ make start-dev ADMIN_ARN=arn:aws:iam::352684330888:role/dev-k8s-01 AUTHENTICATOR_IMAGE=aws-iam-authenticator:v0.5.3_02a86a549cee91b37baff12
d2528f185594fb98c_20220117T003254Z                                                                                                                                                           
make /home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator/_output/bin/aws-iam-authenticator        
make[1]: Entering directory '/home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator'                 
make[1]: '/home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator/_output/bin/aws-iam-authenticator' is up to date.
make[1]: Leaving directory '/home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator'                  
./hack/start-dev-env.sh                                                                             
Creating network authenticator-dev                                                                  
644eb11eaf9b459d7702b1463bf01b66988077fb7119aa1ee5fb0b6e3b109e5e                                                                                                                             
9abc964c717778714e3aa8208abb01c3d67eeb3c6a7ffe6e5689c62b1ff7d15c
Authenticator running at 172.30.0.10                                                                                                                                                         
Creating cluster "authenticator-dev-cluster" ...                                                                                                                                             
WARNING: Overriding docker network due to KIND_EXPERIMENTAL_DOCKER_NETWORK                                                                                                                   
WARNING: Here be dragons! This is not supported currently.    
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼                                                                                                                                              
 ✓ Preparing nodes 📦                                                                                                                                                                        
 ✓ Writing configuration 📜                                
 ✓ Starting control-plane 🕹️                              
 ✓ Installing CNI 🔌                                                                                                                                                                         
 ✓ Installing StorageClass 💾                                                                                                                                                                
Set kubectl context to "kind-authenticator-dev-cluster"                                                                                                                                      
You can now use your cluster with:                                                                                                                                                           
                                                                                                                   
kubectl cluster-info --context kind-authenticator-dev-cluster --kubeconfig /home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator/_output/dev/client/kind-kubeconfig.yaml
                                                                     
Thanks for using kind! 😊                                   
.                                                                                                       
Test authenticator with:                                                               
kubectl --kubeconfig="/home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator/_output/dev/client/kubeconfig.yaml" --context="test-authenticator"
.

And the authenticator client and server are exercised with the following parameters:

$ kubectl --kubeconfig="/home/ubuntu/go/src/sigs.k8s.io/aws-iam-authenticator/_output/dev/client/kubeconfig.yaml" --context="test-authenticator" get pods                                                                                                                                                                                 
No resources found in default namespace.

- Scripts to create and tear down the environment. - Uses a kind cluster and a stand-alone authenticator container. - Sets up a kubeconfig to test client-server interactions end to end.

k8s-ci-robot · 2022-01-18T02:26:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nckturner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nckturner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nckturner · 2022-01-18T17:31:48Z

/retest

jaypipes

@nckturner this is a great start. I have some ideas inline that might simplify things, however.

jaypipes · 2022-01-18T14:37:08Z

hack/lib/dev-env.sh

+# Parameters
+
+# Parameters required to be set by caller for dev environment creation only:
+#   AUTHENTICATOR_IMAGE


how about just defaulting to building an image of the locally checked-out code?

jaypipes · 2022-01-18T23:15:40Z

hack/lib/dev-env.sh

+# Check that required binaries are installed
+command -v make >/dev/null 2>&1 || { echo >&2 "make is required but it's not installed.  Aborting."; exit 1; }
+command -v docker >/dev/null 2>&1 || { echo >&2 "docker is required but it's not installed.  Aborting."; exit 1; }
+command -v kind >/dev/null 2>&1 || { echo >&2 "kind is required but it's not installed.  Aborting."; exit 1; }


this is fine, but consider having a check that validates a particular version of these dependencies. We've found in ACK land that just relying on docker or kind (and not a specific version of those) leads to ambiguous dev environments and lots of headache

Hm, that's a good point. At least printing a warning with expected/tested versions.

jaypipes · 2022-01-18T23:17:57Z

Makefile

+
+# Use make start-dev when you want to create a kind cluster for
+# testing the authenticator.  You must pass in the admin ARN and
+# the image under test.


See note below. Consider defaulting the image to one built from the locally checked-out source.

jaypipes · 2022-01-18T23:19:28Z

hack/lib/dev-env.sh

+# Admin kubeconfig generated by kind
+kind_kubeconfig="${client_dir}/kind-kubeconfig.yaml"
+
+function create_network() {


is there a requirement to create a separate virtual network? Why not use the default docker virtual network?

Kind already creates/uses its own network, so in order to include the authenticator in that network I needed to do it this way (or run authenticator on the kind node along with the apiserver, etc, but in that case I would have had to do some setup slightly differently, including generating all config/certs up front instead of letting authenticator generate them. This is because the ca.cert needs to embedded in a webhook kubeconfig and passed to the apiserver on startup). Also interacting with the apiserver as a standalone docker container is quite easy, and we don't have to interact with kind for debugging... e.g. port-forwarding in order to curl the metrics endpoint, etc.

jaypipes · 2022-01-18T23:25:40Z

hack/lib/dev-env.sh

+function start_authenticator() {
+    mkdir -p "${authenticator_state_host_dir}"
+    mkdir -p "${authenticator_export_host_dir}"
+    chmod 777 "${authenticator_state_host_dir}"
+    chmod 777 "${authenticator_export_host_dir}"
+    docker run \
+        --detach \
+        --ip "${AUTHENTICATOR_IP}" \
+        --mount "type=bind,src=${authenticator_config_host_dir},dst=${authenticator_config_dest_dir}" \
+        --mount "type=bind,src=${authenticator_state_host_dir},dst=${authenticator_state_dest_dir}" \
+        --mount "type=bind,src=${authenticator_export_host_dir},dst=${authenticator_export_dest_dir}" \
+        --name aws-iam-authenticator \
+        --network "${NETWORK_NAME}" \
+        --publish ${authenticator_healthz_port}:${authenticator_healthz_port} \
+        --publish ${AUTHENTICATOR_PORT}:${AUTHENTICATOR_PORT} \
+        --rm \
+        "${AUTHENTICATOR_IMAGE}" \
+        server \
+        --config "${authenticator_config_dest_dir}/authenticator.yaml"
+}
+
+function kill_authenticator() {
+    docker kill aws-iam-authenticator || true
+}


the issue I have with this is that it's not actually testing the installation/operating practice for aws-iam-authenticator that is described in the README: deploy the authenticator as a Daemonset on each control plane node in the k8s cluster, using the example manifest. Why not just kubectl apply a slightly-modified version of the example YAML files against the KinD cluster? That would do dual service of validating the recommended installation approach for the authenticator.

Yeah. The reason I went with the above approach is because I decided to let the authenticator generate its own certificate and the webhook kubeconfig on startup. Since the path to this config needs to be passed to the apiserver as a flag on startup, this suggests starting the authenticator before the kind cluster, because the API server is started automatically on cluster creation and is configured in the kind cluster configuration. That being said, we can just use the init command to pre-generate the configuration, and then launch authenticator, and then patch the API server.

We should try the approach that allows quicker iteration that does not need a tear down and restart of the kind cluster. In both approaches i think we get it, probably bit easier with daemonset approach.
The second criteria for choosing is closeness to setups. I'm not sure which one is more prevalent, but i'd lean towards the approach implemented due to my familiarity with managed setups.

jyotimahapatra · 2022-01-19T02:20:14Z

hack/lib/dev-env.sh

+
+function write_kind_config() {
+    mkdir -p "${kind_config_host_dir}"
+    sed -e "s|{{AUTHENTICATOR_EXPORT_HOST_DIR}}|${authenticator_export_host_dir}|g" \


do these commands work on mac? i know sed has different syntax for mac and linux. We can mandate installation of gsed to keep these working across both platforms?

Yeah, I have no idea if they work on Mac but I've definitely ran into problems with sed commands that were written by mac users :)

jaypipes · 2022-01-19T18:41:49Z

Chatted about this offline. Good with improvements in followup PRs :)

/lgtm

Add kind based development environment

cdd70af

- Scripts to create and tear down the environment. - Uses a kind cluster and a stand-alone authenticator container. - Sets up a kubeconfig to test client-server interactions end to end.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 18, 2022

k8s-ci-robot requested review from jaypipes and justinsb January 18, 2022 02:26

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 18, 2022

Publish ports in dev env

e8fc28f

jaypipes suggested changes Jan 18, 2022

View reviewed changes

jyotimahapatra reviewed Jan 19, 2022

View reviewed changes

k8s-ci-robot assigned jaypipes Jan 19, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 19, 2022

k8s-ci-robot merged commit f974d5e into kubernetes-sigs:master Jan 19, 2022

jyotimahapatra mentioned this pull request Feb 2, 2022

[1/2] Release 0.5.4 #425

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kind-based development environment #422

Add kind-based development environment #422

nckturner commented Jan 18, 2022

k8s-ci-robot commented Jan 18, 2022

nckturner commented Jan 18, 2022

jaypipes left a comment

jaypipes Jan 18, 2022

jaypipes Jan 18, 2022

nckturner Jan 19, 2022

jaypipes Jan 18, 2022

jaypipes Jan 18, 2022

nckturner Jan 19, 2022

jaypipes Jan 18, 2022

nckturner Jan 19, 2022

jyotimahapatra Jan 19, 2022

jyotimahapatra Jan 19, 2022 •

edited

nckturner Jan 20, 2022

jaypipes commented Jan 19, 2022

Add kind-based development environment #422

Add kind-based development environment #422

Conversation

nckturner commented Jan 18, 2022

k8s-ci-robot commented Jan 18, 2022

nckturner commented Jan 18, 2022

jaypipes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyotimahapatra Jan 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaypipes commented Jan 19, 2022

jyotimahapatra Jan 19, 2022 •

edited