Tekton Comparison Demo: Jenkins vs. Tekton

Table of Contents

• Introduction
• Demo Setup
  • Slackbot Setup
    • Webhooks
    • Interactivity
    • Authentication
  • Demo Installation
• Troubleshooting
  • MySQL
  • Troubleshooting Pipeline Tasks
    • General
    • Volume Issues
  • Troubleshooting OpenShift Permissions
  • Troubleshooting (Local) Tomcat Server
  • OpenShift Pipeline (Git) Triggers
    • Triggered Pipeline Fails to Run
    • Viewing (Extended) OpenShift Pipeline (Tekton) Logs
• Appendix
  • More about OpenShift Pipelines (Tekton)
  • Slack Interceptor

Introduction

This repo contains all the scripts and code (as submodules) needed to create a demo that supports this walkthrough that compares and contrasts traditional Jenkins pipelines to Tekton. Concepts covered include:

Jenkins	OpenShift Pipelines
Blue Ocean	Pipeline Console
OpenShift Plugin / Kubernetes PodTemplate
Webhooks	Tekton Events
S2I binary integration	S2I binary integration

Tip	There is a .devcontainer in this repo so if you have VisualStudio Code with remote extensions enabled, you can open repo folder in a container that has all the tools necessary to run this demo already installed.

Demo Setup

Before installing the demo you need to create a Slack webhook for Tekton to post notifications to.

• If you’ve done this before and have a webhook URL, you can skip to the demo installation section with that URL

• If you’ve not done this already, see the next section for details on how to create one

Slackbot Setup

The CI/CD toolchain that is setup by the create demo script will prompt to slack when a deployment to dev is completed. In order for this to work, you need to:

1. setup a slack app (bot) and obtain a bot specific webhook to be able to post.

2. Then associate that bot with a channel so that you can see the posts

3. Setup interactivity in the bot so that it can post back to your cluster when you click the button on the message

Apps can be created from here

Webhooks

You will need a webhook in order to call your app/bot from outside. This can be generated by going to the "Incoming Webhooks" part of the app

It is this highlighted hook that you pass into the create-demo.sh

Interactivity

The Tekton pipeline will be setup to call out to your bot. Assuming your bot is associated with a channel, you’ll see something like the following posted:

In order to get the highlighted button to do something you must update the Request URL untio the bot’s Interactivity and Shortcuts to point to the external route for the slack-webhook.

You can get the route to the webhook by issuing this command:

echo "http://$(oc get route webhook-slack -n ${PROJECT_PREFIX}-cicd -o jsonpath='{.spec.host}')/"

Then turn interactivity on and paste that route in the slot indicated

Authentication

Once you’ve created an app, there will be a number of authentication based creds that will be generated

Demo Installation

Note	The script referenced is run during the Walkthrough. This just gives a little more info if needed for troubleshooting

A good example on how to get this running with SpringBoot is here. This demo is heavily based on it. this is also a good article about a number of Tekton features used in this demo (such as workspaces) including some others that aren’t yet being used (e.g. conditions)

Run the following script to setup the entire demo which includes a Tekton as well as a Jenkins pipeline for building a petclinic tomcat app. Pipelines will be created a project called $PROJECT_PREFIX-cicd (where $PROJECT_PREFIX is as defined below). The project will be created if it doesn’t exist already.

Finally, to minimize compatibility issues when installing from different OSes, the installation process should be run from inside a container.

1. If you are not using VSCode Remove Containers then follow these steps to get a container running in which you’ll issue all the following commands

   1. Change to the root of this repository

   2. Run the following command

      docker run -it -v ~/.kube:/home/jboss/.kube -v ~/.oh-my-zsh:/home/jboss/.oh-my-zsh -v $(pwd):/workspaces/tekton-comparison-demo -w /workspaces/tekton-comparison-demo quay.io/mhildenb/tekton-demo-shell:latest /bin/zsh

2. Setup the following variables in your environment (or in the currently running shell):

   • PROJECT_PREFIX this is set to petclinic by default, but you can override this to change the names of projects created

   • SLACK_WEBHOOK this is the webhook url to be used for triggering a "prompt to staging?" message to slack

   • USER this should be set to your RedHat access user name (used to create a pull secret for registry.redhat.io)

   • PASSWORD this should be set to your RedHat access password (used to create a pull secret for registry.redhat.io)

3. Run the following command to install the whole demo

   • NOTE: The -i flag indicates that all pre-requisites should be installed (such as OpenShift Pipelines)

$DEMO_HOME/scripts/create-tekton-cicd.sh install --project-prefix $PROJECT_PREFIX -i --user $USER --password "$PASSWORD" --slack-webhook-url "$SLACK_WEBHOOK"

Troubleshooting

MySQL

You can test access to a MySQL database in an OpenShift cluster using the Adminer image.

1. First, setup port forwarding to the service in question (assuming a petclinic based service as shown in the walkthrough)

   oc port-forward svc/petclinic-mysql 3306:3306

2. Then, in another shell, run the Adminer image and have it port forward to 8080. NOTE: Assumes you are running on a Mac using Docker for Mac, this is where the docker.for.mac.localhost stuff comes from

   docker run -p 8080:8080 -e ADMINER_DEFAULT_SERVER=docker.for.mac.localhost adminer:latest

3. From the Adminer web page, login as root (using whatever secret was used in the setup of the cluster). You can then run arbitrary commands. Here are the commands you can run to grant access to a user pc to a newly created petclinic database (from here)

   CREATE USER 'pc'@'%' IDENTIFIED BY 'petclinic';
   CREATE DATABASE petclinic;
   GRANT ALL PRIVILEGES ON petclinic.* TO 'pc'@'%';

   1. Or instead, you run SQL commands from the local command line

      oc run mysql-client --image=mysql:5.7 --restart=Never --rm=true --attach=true --wait=true \
          -- mysql -h petclinic-mysql -uroot -ppetclinic -e "CREATE USER 'pc'@'%' IDENTIFIED BY 'petclinic'; \
            CREATE DATABASE petclinic; \
            GRANT ALL PRIVILEGES ON petclinic.* TO 'pc'@'%';"

Troubleshooting Pipeline Tasks

General

If a pipeline fails and the logs are not enough to determine the problem, you can use the fact that every task maps to a pod to your advantage.

Let’s say that the task "unit-test" failed in a recent run.

1. First look for the pod that represents that run

   $ oc get pods
   NAME                                                              READY   STATUS      RESTARTS   AGE
   petclinic-dev-pipeline-tomcat-dwjk4-checkout-vnp7v-pod-f8b5j      0/1     Completed   0          3m18s
   petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46    0/1     Error       0          3m
   petclinic-dev-pipeline-tomcat-kpbx9-checkout-t78sr-pod-qnfrh      0/1     Error       0

2. Then use the oc debug command to restart that pod to look around:

   $ oc debug po/petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46
   Starting pod/petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46-debug, command was: /tekton/tools/entrypoint -wait_file /tekton/downward/ready -wait_file_content -post_file /tekton/tools/0 -termination_path /tekton/termination -entrypoint ./mvnw -- -Dmaven.repo.local=/workspace/source/artefacts -s /var/config/settings.xml package
   If you don't see a command prompt, try pressing enter.
   sh-4.2$

Volume Issues

Sometimes pipelines fail to run because the workspace volume cannot be mounted. Looks like to root cause has to do with the underlying infra volume being deleted out from underneath a PersistentVolume. If you have pipelines that are timing out due to pods failing to run (usually you won’t get any log stream), take a look at the events on the pod and see if you notice these kind of mounting errors:

This can usually be remedied by deleting the PVC, but finalizers keep PVCs from being deleted if a pod has a claim.

If you run into this issue, cancel the affected pipeline (otherwise the PVC won’t be able to be deleted) and either run the following command or see the additional details that follow

scripts/util-recreate-pvc.sh pipeline-source-pvc.yaml

To see all the claims on a PVC, look for the Mounted By section of the output of the following describe command (for pvc/maven-source-pvc):

oc describe pvc/maven-source-pvc

To delete all pods that have a claim on the pvc pvc/maven-source-pvc:

oc delete pods $(oc describe pvc/maven-source-pvc | grep "Mounted By" -A40 | sed "s/ //ig" | sed "s/MountedBy://ig")

Troubleshooting OpenShift Permissions

You can use the oc run command to run certain containers in a given project as a service account.

For instance, this command can be used to see what kind of permissions the builder service account has to view other projects (e.g. access to remote imagestreams)

oc run test3 --image=quay.io/openshift/origin-cli:latest --serviceaccount=builder -it --rm=true

Troubleshooting (Local) Tomcat Server

If the tomcat extension fails to run, you can attempt the following:

1. remote any old tomcat files

   rm -f /opt/webserver/webse*

2. Attempt to readd tomcat to /opt/webserver per the instructions above

3. if that still doesn’t work, rebuild container.

4. If all else fails, you can run the tomcat server locally.

OpenShift Pipeline (Git) Triggers

Tekton allows for EventListeners, TriggerTemplates, and TriggerBindings to allow a git repo to hit a webhook and trigger a build. See also here. To get basic triggers going for both gogs and github run the following:

Note	For an example of triggers working with Tekton, see files in the template directory of this repo

Note	You may also want to consider this tekton dashboard functionality

YAML resources for the pipeline created for this demo can be found in these locations:

1. Resources: $DEMO_HOME/kube/tekton/resources

2. Triggers: $DEMO_HOME/kube/tekton/triggers

Triggered Pipeline Fails to Run

If the trigger doesn’t appear to fire, then check the logs of the pod that is running that represents the webhook. The probably is likely in the PipelineRun template.

Viewing (Extended) OpenShift Pipeline (Tekton) Logs

You can see limited logs in the Tekton UI, but if you want the full logs, you can access these from the command line using the tkn command

# Get the list of pipelineruns in the current project
tkn pipelinerun list

# Output the full logs of the named pipeline run (where petclinic-deploy-dev-run-j7ktj is a pipeline run name )
tkn pipelinerun logs petclinic-deploy-dev-run-j7ktj

To output the logs of a currently running pipelinerun (pr) and follow them, use:

tkn pr logs -L -f

Appendix

More about OpenShift Pipelines (Tekton)

Pipeline setup is inspired by this post here

From the instructions here

1. Install subscription (in openshift operators)

   oc apply -f $DEMO_HOME/kube/tekton/tekton-subscription.yaml

2. Optionally install tekton dashboard (for visualization) as per here

   oc apply -f $DEMO_HOME/kube/tekton/openshift-tekton-dashboard-release.yaml --validate=false
   oc wait --for=condition=Available deployment/tekton-dashboard -n openshift-pipelines

3. Then you can open the dashboard by hitting this URL. It will authenticate using OpenShift oauth

   echo "https://$(oc get route tekton-dashboard -o jsonpath='{.spec.host}' -n openshift-pipelines)/"

4. When the operator has finished installing, it will install a pipeline service account in all projects that have sufficient permissions to build stuff. There is also a centralized openshift-pipelines project that holds pipeline supporting pods.

   Note	See also tips and tricks from the pipelines tutorial

Slack Interceptor

The Slack integration that you find in this webhook includes a webhook Interceptor that translates the form urlencoded output of the slack callback into a json document.

In this demo this is deployed as a service that points to the container quay.io/mhildenb/slack-webhook-interceptor:latest. You can find the code that backs that container here. This is itself a fork of this repo.