Airflow on Kubernetes

Run Airflow on Kubernetes. This repository contains scripts to;

1. run a multi-node kubernets cluster on local machine using KinD
2. prepare Dockerfile for your airflow dags
3. minimal helm chart required to run airflow on kubernetes using Kubernetes Executor

Watch video tutorial here https://youtu.be/RqSYh3UI_Is

Prerequisites

Prior knowledge on Airflow as well as Kubernetes is a must.

Airflow tutorials: https://www.youtube.com/playlist?list=PLzKRcZrsJN_xcKKyKn18K7sWu5TTtdywh

Articles:

1. https://maxcotec.com/apache-airflow-architecture
2. https://maxcotec.com/blog/apache-airflow-2-docker-beginners-guide/

Requirements

kind: https://kind.sigs.k8s.io/docs/user/quick-start/#installation

kubectl: https://kubernetes.io/docs/tasks/tools

k9s: https://k9scli.io/topics/install

docker: https://docs.docker.com/get-docker

Tools version used in this tutorial:

kind: v0.14.0

k9s: 0.25.18

kubectl: Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}

Docker desktop: 4.10.1

Run Instructions

1. create kubernets cluster. Run bash script create_cluster_with_registry.sh from directory kind_cluster/
2. write your own dag and place it inside airflow-dags/dags directory. Then build docker file docker build . -t airflow_dags:0.1.0
3. tag the image with local docker registry host docker tag airflow_dags:0.1.0 localhost:5001/airflow_dags:0.1.0
4. push the airflow image to local docker registry docker push localhost:5001/airflow_dags:0.1.0
5. deploy helm chart helm upgrade --install airflow . --values values.yaml --set airflow.dags_image.repository=localhost:5001/airflow_dags --set airflow.dags_image.tags=0.1.0

⚡ Quick start

1. Create Kubernetes cluster: kind_cluster/create_cluster_with_registry.sh from root directory
2. Setup Airflow on the cluster: ./setup_cluster.sh from the root directory
3. Port-forwarding to Airflow: kubectl port-forward svc/airflow-webserver-svc 8080:8080 -n default
4. To delete cluster: ./delete_cluster.sh

KubernetesPodOperator DAG

A new DAG has been added named hello_KubePodOp (located here airflow-dags/dags/kubePodOpDag.py), that demonstrates usage of KubernetesPodOperator as a task. For demonstration purpose, we use demo python code from this repository https://github.com/maxcotec/k8sPodOp-app. Build image by following instructions on repo README and push the image to local docker registry. You can also use your own docker image to run against KubernetesPodOperator.

Video tutorials; part 1: https://youtu.be/b1gpbGB058M part 2: https://youtu.be/2JFFzTw0a-U