The osdu - Kubernetes Architecture solution template is intended to provision Managed Kubernetes resources like AKS and other core OSDU cloud managed services like Cosmos, Blob Storage and Keyvault.
We decided to split these configuration files out into a separate Terraform environment templates to mitigate the risk of Terraform accidentally deleting stateful resources types as well as have a mechanism to duplicate environments to support concepts such as data partitioning or multiple AKS Clusters.
Technical design specifications
GitOps design specifications.
Cloud administrators who are versed with both Cobalt templating and Kubernetes.
- Azure Subscription
- An available Service Principal with API Permissions granted with Admin Consent within Azure app registration. The required Azure Active Directory Graph app role is
Application.ReadWrite.OwnedBy
- Terraform and Go are locally installed
- Azure Storage Account is setup to store Terraform state
- Local environment variables are setup
- Deployment Service Principal is granted Owner level role assignment for the target Azure subscription
- Enroll as an Azure subscriber. The free trial subscription does not support enough cores to run this tutorial.
- Terraform
common_resourcesenvironment module is provisoned to your Azure Environment - Terraform
data_resourcesenvironment module is provisoned to your Azure Environment - Install the required common tools (kubectl, helm, and terraform). See also Required Tools. Note: this tutorial currently uses Terraform 0.12.28.
This document assumes one is running a current version of Ubuntu. Windows users can install the Ubuntu Terminal from the Microsoft Store. The Ubuntu Terminal enables Linux command-line utilities, including bash, ssh, and git that will be useful for the following deployment. Note: You will need the Windows Subsystem for Linux installed to use the Ubuntu Terminal on Windows.
Ensure that the required tools, are installed in your environment. Alternatively, there are scripts that will install helm, terraform and kubectl. In this case, use setup_kubernetes_tools.sh and setup_terraform.sh. The scripts install the tools into /usr/local/bin.
For information specific to your operating system, see the Azure CLI install guide. You can also use this script if running on a Unix based machine.
We will deploy the Bedrock environment using the empty repo and then add a Kubernetes manifest that defines a simple Web application. The change to the repo will automatically update the deployment.
To prepare the Flux manifest repository, we must:
- Create the Flux Manifest Repository
- Generate an RSA Key Pair to use as the Manifest Repository Deploy Key
- Grant Deploy Key access to the Manifest Repository
Create an empty git repository with a name that clearly signals that the repo is used for the Flux manifests. For example bedrock-deploy-demo.
Flux requires that the git repository have at least one commit. Initialize the repo with an empty commit.
git commit --allow-empty -m "Initializing the Flux Manifest Repository"When you are deploying services, you will put the manifests in providers/azure/hld-registry directory of this repo. Once you push the manifests, Flux should automatically deploy the services.
Generate the deploy key using ssh-keygen. The public portion of the key pair will be uploaded to GitHub as a deploy key.
AZURE_VAULT=<common_vault_name>
KEY_NAME=gitops-ssh-key
# Generate gitops-ssh-key
ssh-keygen -b 4096 -t rsa -f $KEY_NAME
# Save gitops-ssh-key
az keyvault secret set --vault-name $AZURE_VAULT -n "${KEY_NAME}" -f "${KEY_NAME}"
az keyvault secret set --vault-name $AZURE_VAULT -n "${KEY_NAME}-pub" -f "${KEY_NAME}.pub"
# Show Public gitops-ssh-key
az keyvault secret show --vault-name $AZURE_VAULT -n "${KEY_NAME}-pub" --query value -otsvThis will create public and private keys for the Flux repository. We will assign the public key under the following heading: Adding the Repository Key. The private key is stored on the machine originating the deployment.
The public key of the RSA key pair previously created needs to be added as a deploy key. Note: If you do not own the repository, you will have to fork it before proceeding.
Use the contents of the Secret as shown above.
Next, in your Azure DevOPS Project, follow these steps to add your public SSH key to your ADO environment.
The Terraform scripts use this node key to setup log-in credentials on the nodes in the AKS cluster. We will use this key when setting up the Terraform deployment variables. Generate the Node Key:
AZURE_VAULT=<common_vault_name>
KEY_NAME=node-ssh-key
# Generate node-ssh-key
ssh-keygen -b 4096 -t rsa -f $KEY_NAME
# Save node-ssh-key
az keyvault secret set --vault-name $AZURE_VAULT -n "${KEY_NAME}" -f "${KEY_NAME}"
az keyvault secret set --vault-name $AZURE_VAULT -n "${KEY_NAME}-pub" -f "${KEY_NAME}.pub"
# Save Locally Public node-ssh-key
az keyvault secret show --vault-name $AZURE_VAULT -n "${KEY_NAME}-pub" --query value -otsvDownload the required keys from the common Key Vault
AZURE_VAULT=<common_vault_name>
az keyvault secret show --vault-name $AZURE_VAULT -n "node-ssh-key-pub" --query value -otsv > ~/.ssh/node-ssh-key.pub
az keyvault secret show --vault-name $AZURE_VAULT -n "gitops-ssh-key" --query value -otsv > ~/.ssh/gitops-ssh-key
chmod 644 ~/.ssh/node-ssh-key.pub
chmod 600 ~/.ssh/gitops-ssh-key
Update your .env file with the paths to your public and private SSH keys for Node and GitOPS repo access.
TF_VAR_ssh_public_key_file=/home/$USER/.ssh/node-ssh-key.pub
TF_VAR_gitops_ssh_key_file=/home/$USER/.ssh/gitops-ssh-key
Follow the directions in here.
Follow the directions in the common_resources environment.
Follow the directions in the data_resources environment.
Follow the directions in the cluster_resources environment.
After terraform apply finishes for the service_resources, there is one critical output artifact: the Kubernetes config file for the deployed cluster that is generated and saved in the output directory. The default file is output/bedrock_kube_config. The following steps use this file to interact with the deployed Bedrock AKS cluster.
Using the config file output/bedrock_kube_config, one of the first things we can do is list all pods deployed within the cluster:
KUBECONFIG=./output/bedrock_kube_config kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
agic agic-ingress-azure-7b88b4b69f-p9n5w 1/1 Running 0 15d
cert-manager jetstack-cert-manager-567bb678c7-sjmct 1/1 Running 0 21h
cert-manager jetstack-cert-manager-cainjector-695d847cdd-l2rv6 1/1 Running 0 21h
cert-manager jetstack-cert-manager-webhook-5b895bb689-dvwvd 1/1 Running 0 21h
flux flux-6899458bb8-qghrq 1/1 Running 8 15d
flux flux-memcached-8647794c5f-slsvr 1/1 Running 0 15d
keda keda-operator-5895ff46b9-fh5xn 1/1 Running 0 3d3h
keda keda-operator-metrics-apiserver-6774776dbc-jwg7q 1/1 Running 0 8d
kube-system azure-cni-networkmonitor-745vs 1/1 Running 0 15d
kube-system azure-cni-networkmonitor-9kq6c 1/1 Running 0 15d
kube-system azure-cni-networkmonitor-dt7ch 1/1 Running 0 15d
kube-system azure-ip-masq-agent-6kv6v 1/1 Running 0 15d
kube-system azure-ip-masq-agent-p6zxn 1/1 Running 0 15d
kube-system azure-ip-masq-agent-vw7fr 1/1 Running 0 15d
kube-system azure-npm-f9qz7 1/1 Running 0 10d
kube-system azure-npm-j6qdv 1/1 Running 0 10d
kube-system azure-npm-vkghz 1/1 Running 0 10d
kube-system coredns-869cb84759-69nmv 1/1 Running 0 15d
kube-system coredns-869cb84759-fvpf8 1/1 Running 0 15d
kube-system coredns-autoscaler-5b867494f-wvt6q 1/1 Running 11 15d
kube-system dashboard-metrics-scraper-7dbbb6996d-4v6m4 1/1 Running 0 15d
kube-system kube-proxy-4qmfg 1/1 Running 0 15d
kube-system kube-proxy-b66qd 1/1 Running 0 15d
kube-system kube-proxy-wwts4 1/1 Running 0 15d
kube-system kubernetes-dashboard-5596bdb9f-cx4bw 1/1 Running 8 15d
kube-system metrics-server-6cd7558856-lvqrg 1/1 Running 0 15d
kube-system omsagent-4g9qf 1/1 Running 0 15d
kube-system omsagent-j8v77 1/1 Running 0 15d
kube-system omsagent-rs-764c6f8d8-54fjd 1/1 Running 0 15d
kube-system omsagent-zg2wh 1/1 Running 0 15d
kube-system tunnelfront-7cfc889c77-gh9jv 2/2 Running 1 15d
kvsecrets kvsecrets-csi-secrets-store-provider-azure-ddnwn 1/1 Running 0 15d
kvsecrets kvsecrets-csi-secrets-store-provider-azure-j2m7x 1/1 Running 0 15d
kvsecrets kvsecrets-csi-secrets-store-provider-azure-sszrt 1/1 Running 0 15d
kvsecrets kvsecrets-secrets-store-csi-driver-92n8k 3/3 Running 0 15d
kvsecrets kvsecrets-secrets-store-csi-driver-pnx8x 3/3 Running 0 15d
kvsecrets kvsecrets-secrets-store-csi-driver-vmg48 3/3 Running 0 15d
osdu default-service-86cd47b748-7mrnw 1/1 Running 0 14d
osdu entitlements-azure-cb59875bc-ncqll 1/1 Running 0 8d
osdu function-debug 1/1 Running 0 3d4h
osdu indexer-7dfcdfbb-bf7sl 1/1 Running 0 30h
osdu legal-57cbd6cd66-tx6sf 1/1 Running 0 8d
osdu legal-debug 1/1 Running 0 9d
osdu search-5f59bc7c85-5wc4l 1/1 Running 0 27h
osdu storage-7d794b54cf-2w5cn 1/1 Running 0 47h
podidentity aad-pod-identity-mic-57dbd9f4fb-bd4vr 1/1 Running 12 15d
podidentity aad-pod-identity-mic-57dbd9f4fb-zj6lb 1/1 Running 0 15d
podidentity aad-pod-identity-nmi-fmjlk 1/1 Running 0 15d
podidentity aad-pod-identity-nmi-qk2t4 1/1 Running 0 15d
podidentity aad-pod-identity-nmi-vtxmg 1/1 Running 0 15dNote that there is also a namespace flux. As previously mentioned, Flux is managing the deployment of all of the resources into the cluster. Taking a look at the description for the flux pod flux-5897d4679b-tckth, we see the following:
$ KUBECONFIG=./output/bedrock_kube_config kubectl describe po/flux-5897d4679b-tckth --namespace=flux
Name: flux-5897d4679b-tckth
Namespace: flux
Priority: 0
PriorityClassName: <none>
Node: aks-default-30249513-2/10.10.1.66
Start Time: Tue, 18 Jun 2019 06:32:49 +0000
Labels: app=flux
pod-template-hash=5897d4679b
release=flux
Annotations: <none>
Status: Running
IP: 10.10.1.76
Controlled By: ReplicaSet/flux-5897d4679b
Containers:
flux:
Container ID: docker://cc4cf38387a883f964cc65b9a1dd13439be756be3cf2d84fa1ca2ced69d98c3a
Image: docker.io/weaveworks/flux:1.12.2
Image ID: docker-pullable://weaveworks/flux@sha256:368bc5b219feffb1fe00c73cd0f1be7754591f86e17f57bc20371ecba62f524f
Port: 3030/TCP
Host Port: 0/TCP
Args:
--ssh-keygen-dir=/var/fluxd/keygen
--k8s-secret-name=flux-ssh
--memcached-hostname=flux-memcached
--memcached-service=
--git-url=git@github.com:jmspring/manifests.git
--git-branch=master
--git-path=prod
--git-user=Weave Flux
--git-email=support@weave.works
--git-set-author=false
--git-poll-interval=5m
--git-timeout=20s
--sync-interval=5m
--git-ci-skip=false
--registry-poll-interval=5m
--registry-rps=200
--registry-burst=125
--registry-trace=false
State: Running
Started: Tue, 18 Jun 2019 06:33:18 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 50m
memory: 64Mi
Environment:
KUBECONFIG: /root/.kubectl/config
Mounts:
/etc/fluxd/ssh from git-key (ro)
/etc/kubernetes/azure.json from acr-credentials (ro)
/root/.kubectl from kubedir (rw)
/var/fluxd/keygen from git-keygen (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flux-token-d2h55 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kubedir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: flux-kube-config
Optional: false
git-key:
Type: Secret (a volume populated by a Secret)
SecretName: flux-ssh
Optional: false
git-keygen:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
acr-credentials:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/azure.json
HostPathType:
flux-token-d2h55:
Type: Secret (a volume populated by a Secret)
SecretName: flux-token-d2h55
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m30s default-scheduler Successfully assigned flux/flux-5897d4679b-tckth to aks-default-30249513-2
Normal Pulling 3m22s kubelet, aks-default-30249513-2 pulling image "docker.io/weaveworks/flux:1.12.2"
Normal Pulled 3m12s kubelet, aks-default-30249513-2 Successfully pulled image "docker.io/weaveworks/flux:1.12.2"
Normal Created 2m57s kubelet, aks-default-30249513-2 Created container
Normal Started 2m57s kubelet, aks-default-30249513-2 Started containerFlux automation makes it easy to upgrade services or infrastructure deployed by Bedrock. In this example Flux watches the repo we set up previously under the heading Set Up Flux Manifest Repository. Now we add a simple Web application to the running deployment by pushing a .yaml manifest to the repo. The .yaml specification describes the service mywebapp and type: a LoadBalancer. It specifies the source the Docker image that contains it: image: andrebriggs/goserver:v1.2 and how many containers to run: replicas: 3. The containers will be accessible through the load balancer.
When the .yaml file is complete we will push it to the repo, or simply drop it on GitHub. Flux is querying the repo for changes and will deploy the new service replicas as defined by this manifest.
Create the following .yaml file and name it something like myWebApp.yaml. The image for this application is specified by the line: image: andrebriggs/goserver:v1.2.
# mywebapp services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: mywebapp
labels:
app: mywebapp
spec:
type: LoadBalancer
ports:
- port: 8080
name: http
selector:
app: mywebapp
---
apiVersion: extensions/v1beta1 #TODO: Migrate to apps/v1
kind: Deployment
metadata:
name: mywebapp-v1
spec:
replicas: 3
minReadySeconds: 10 # Wait 2 seconds after each new pod comes up before marked as "ready"
strategy:
type: RollingUpdate # describe how we do rolling updates
rollingUpdate:
maxUnavailable: 1 # When updating take one pod down at a time
maxSurge: 1 # When updating never have more than one extra pod. If replicas = 2 then never 3 pods when updating
template:
metadata:
labels:
app: mywebapp
version: v1
spec:
containers:
- name: mywebapp
image: andrebriggs/goserver:v1.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
---
To see the changes as Flux picks them up and deploys them, open a bash command window and navigate to the bedrock/cluster/environments/azure-simple directory.
Get your Flux pod name by running: KUBECONFIG=./output/bedrock_kube_config kubectl get pod -n flux
Copy the name of the pod (the one that is not memcached).
Then run the command: KUBECONFIG=./output/bedrock_kube_config kubectl logs -f <Flux-pod-name> --namespace=flux. This will display a running log of the deployment.
Now, push or drop the myWebApp.yaml file to the empty repo created under the previous heading Set Up Flux Manifest Repository. You can click Upload files on the GitHub repo page and drop the .yaml file:
In this output, Flux has found the repo bedrock-deploy-demo and created the new service: "kubectl apply -f -" took=1.263687361s err=null output="service/mywebapp created\ndeployment.extensions/mywebapp-v1 created".
Open another bash window. When the new service is running, use KUBECONFIG=./output/bedrock_kube_config kubectl get po --all-namespaces to find the new namespaces in the deployment.
Then run KUBECONFIG=./output/bedrock_kube_config kubectl get svc --all-namespaces. The output will include the EXTERNAL-IP address and PORT of the mywebapp load balancer:
$ KUBECONFIG=./output/bedrock_kube_config kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 44m
default mywebapp LoadBalancer 10.0.96.208 http://168.61.208.215/ 8080:30197/TCP 23m
flux flux ClusterIP 10.0.139.133 <none> 3030/TCP 34m
flux flux-memcached ClusterIP 10.0.246.230 <none> 11211/TCP 34m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 44m
kube-system kubernetes-dashboard ClusterIP 10.0.222.104 <none> 80/TCP 44m
kube-system metrics-server ClusterIP 10.0.189.185 <none> 443/TCP 44mThe EXTERNAL-IP, in this case is: http://168.61.208.215/. Append the port and use http://http://168.61.208.215:8080 to run the service in a browser.
There are six OSDU services that can be deployed in this infrastructure: storage, search, indexer-serivce, indexer-queue, entitlements, and legal. The helm charts for these service can be found in their respective repos at /devops/azure:
You can download the config file for these templates here: wget https://raw.githubusercontent.com/Azure/osdu-infrastructure/master/devops/helm-config.yaml -O config.yaml
To install the services using Flux, you will want a directory stucture like the following:
project root
└───osdu-infrastructure (this repo)
└───osdu-services (folder you create)
| └───entitlements-azure (service repo you will clone)
| └───legal (service repo you will clone)
| └───storage (service repo you will clone)
| └───indexer-service (service repo you will clone)
| └───indexer-queue (service repo you will clone)
| └───search-service (service repo you will clone)
└───flux-repo (the repo for flux manifests you created earlier)
You can then use this script to generate the manifests in your flux repo after filling out your config.yaml file:
SRC_DIR="<ROOT_PATH_TO_SOURCE>" # $HOME/source/osdu
FLUX_SRC="<FULL_PATH_TO_SOURCE>" # $SRC_DIR/flux-repo
INFRA_SRC="<FULL_PATH_TO_SOURCE>" # $SRC_DIR/osdu-infrastructure
SERVICES_DIR="<FULL_PATH_TO_SOURCE>" # $SRC_DIR/osdu-services
BRANCH="master"
TAG="latest"
# Extract manifests from the common osdu chart.
helm template osdu-flux ${INFRA_SRC}/devops/charts/osdu-common -f ${INFRA_SRC}/devops/config.yaml > ${FLUX_SRC}/providers/azure/hld-registry/azure-common.yaml
# Extract manifests from each service chart.
for SERVICE in entitlements-azure legal storage indexer-service indexer-queue search-service ;
do
helm template $SERVICE ${SERVICES_DIR}/$SERVICE/devops/azure/chart --set image.branch=$BRANCH --set image.tag=$TAG > ${FLUX_SRC}/providers/azure/hld-registry/$SERVICE.yaml
doneAfter running this script, verify that the manifests are in your flux repo in the hld-registry directory and that the azure-common manifest is there as well. After verifying that these manifests are all in place, you can deploy the service by pushing the flux repo with the new files.
Kubernetes includes a web dashboard that can be used for basic management operations. This dashboard lets you view basic health status and metrics for your applications, create and deploy services, and edit existing applications.
$ kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
$ az aks browse --resource-group myResourceGroup --name myAKSCluster
The behavior of this command has been altered by the following extension: aks-preview
Merged "devint-aks-mgf9wjxt-osdu-r2-aks" as current context in /tmp/tmps6_a6amm
Proxy running on http://127.0.0.1:8001/
Press CTRL+C to close the tunnel...
Copyright © Microsoft Corporation
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.