Do you know that Elastic performed 21 releases in 2020?
Each time a release is promoted, this is 500+ artifacts published to multiple public places (bucket, Docker registries, Maven Central, Rubygems, and so on) and available on Cloud at the same time. This complex process became a non-event thanks to our Unified Release workflow based on Tekton Tasks and Pipelines and monitored with Elastic Observability.
This blog post shows how to run your first Tekton Task, and then how to install and use the Elastic Observability Solution to monitor many Tasks and Pipelines deployed within a cluster. All you need to have is a Kubernetes cluster available locally and 10 minutes of your time. Elastic Cloud on Kubernetes (ECK) is used to set up the Elastic components.
Note: the examples are available in the tekton-pipelines-elastic-o11y GitHub repository
What’s Tekton Pipelines?
Tekton is an open-source framework for creating CI/CD systems. Tekton Pipelines are the building blocks of Tekton CI/CD workflows. It’s a Kubernetes extension that defines a set of Kubernetes Custom resources (CRDs) from which the workflows are assembled.
The main entities are the following:
-
Step: the most basic building block, this is a Kubernetes container specification -
Task: a collection of steps that run in sequential order in the same Kubernetes node -
TaskRun: to instantiate and execute a Task on a Kubernetes cluster -
Pipeline: combine Tasks together, that run sequentially or concurrently -
PipelineRun: to instantiate and execute a Pipeline on a Kubernetes cluster
Checkout the official website at tektoncd.dev/docs to have more information.
Notes: To keep it simple, this blog post focuses on deploying and running Tasks and TaskRuns, and it does not introduce Pipeline and PipelineRuns resources.
Get your first Tekton Task running!
Now that you know what Tekton Pipelines is, let’s see how to execute a Task. The first steps are to install Tekton Pipelines to your local cluster and then install the Tekton CLI to interact with it.
Install Tekton Pipelines and Tekton CLI (tkn)
Installing Tekton Pipelines is done through a single command line
$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml
You can refer to the official documentation to set up the CLI for your specific environment. For example, for macOS users, tkn is available via brew:
$ brew install tektoncd-cli
Make sure both components are successfully installed by running:
$ tkn version
Client version: 0.14.0
Pipeline version: v0.18.1
Execute your first Task
The goal of this first Task is to checkout the source code from a git repository to your Kubernetes cluster, so you are able to run the tests from this project afterwards.
The commands below install the git-clone Task into the cluster from the Tekton Catalog and create a PersistentVolumeClaim (PVC) to store the source code and share it across multiple Tasks.
# Install the git-clone Task from Tekton Catalog
$ tkn hub get task git-clone --version 0.2 | kubectl apply -f -
task.tekton.dev/git-clone created
# Create a PVC to share data between tasks
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: go-source
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
EOF
persistentvolumeclaim/go-source created
Everything is in place to execute a Task through the deployment of a first TaskRun that is going to invoke the Task:
# Execute a Task to clone the project
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: clone-my-code
spec:
taskRef:
name: git-clone
workspaces:
- name: output
persistentVolumeClaim:
claimName: go-source
params:
- name: url
value: https://github.com/elastic/go-licenser.git
- name: revision
value: master
EOF
taskrun.tekton.dev/clone-my-code created
Great, the TaskRun is deployed and you can check the result. You can also see the logs through the CLI commands:
$ tkn taskruns list
NAME STARTED DURATION STATUS
clone-my-code 1 minute ago 9 seconds Succeeded
$ tkn taskruns logs clone-my-code
...
[clone] + /ko-app/git-init -url https://github.com/elastic/go-licenser.git -revision v0.3.1 -refspec -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[clone] {"level":"info","ts":1607989263.4070501,"caller":"git/git.go:165","msg":"Successfully cloned https://github.com/elastic/go-licenser.git @ 857b4969bc2f753ffb9eb3a885d01a59a9f22cdb (grafted, HEAD) in path /workspace/output/"}
[clone] ...
So far, you have executed a first Task, and checked the output using the CLI.
Next, before executing the Task for running the tests of this project, let’s install the Elastic Observability Solution.
Setup Elastic Observability with Elastic Cloud on Kubernetes
Elastic Cloud on Kubernetes (ECK) is the official operator for provisioning Elastic Stack deployments in Kubernetes. This article is not going to explain how it works. To know more about it, checkout the dedicated blog post Elastic Stack Monitoring with Elastic Cloud on Kubernetes
Observability with the Elastic Stack allows unifying all your logs, metrics, and application trace information in one place. Take a look at the Observability with the Elastic Stack blog post to know more about it.
Install ECK
Installing ECK is done through a single command line:
$ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml
Install Elastic Observability
Installing the Elastic Observability Solution is done in two steps here. First, Elasticsearch and Kibana are deployed, then Metricbeat and Filebeat.
# Deploy Elasticsearch and Kibana
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-es-kb.yaml
# Deploy Metricbeat and Filebeat
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-filebeat-metricbeat.yaml
Check that everything is up and running (wait until filebeat and metricbeat are running for more than 3 minutes):
# Check Elastic pods
$ kubectl get pods -n elastic-system
NAME READY STATUS RESTARTS AGE
elastic-operator-0 1/1 Running 1 5m
elasticsearch-monitoring-es-default-0 1/1 Running 1 5m
filebeat-beat-filebeat-b6vlt 1/1 Running 7 5m
kibana-monitoring-kb-599698987-7cjqq 1/1 Running 1 5m
metricbeat-beat-metricbeat-fsz5p 1/1 Running 7 5m
Get the password for the elastic user needed to access the UI:
# Password for the elastic user
$ echo $(kubectl get secret -n elastic-system elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
XXXXXXXXXXXXXX
Make Kibana available on port 5601:
$ kubectl port-forward -n elastic-system svc/kibana-monitoring-kb-http 5601
Now, you can access Elastic Observability (use the above credentials):
Note: the intent of this tutorial is to provide a development environment. To install a valid certificate, please refer to the official Elastic documentation.
Go to the Metrics UI, in the Inventory view, and click on Show the Kubernetes Pods. Then select Grouped by Namespace. You should see the Pods running in the elastic-system namespace and the ones running in the tekton-pipelines namespace.
Let's execute more Tasks and see how it goes.
Monitor Tasks and Pipelines with Elastic Observability
Run the tests Task and check the logs via the Logs UI
In this step, we are going to execute a Task and automatically see the logs via the UI.
# Install the generic golang-test Task from the catalog
$ tkn hub get task golang-test --version 0.1 | kubectl apply -f -
task.tekton.dev/golang-test created
# Execute a Task to run the tests
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: test-my-code
spec:
taskRef:
name: golang-test
workspaces:
- name: source
persistentVolumeClaim:
claimName: go-source
params:
- name: package
value: github.com/elastic/go-licenser
- name: packages
value: ./...
- name: flags
value: -timeout 10s -p 4 -race -cover
EOF
taskrun.tekton.dev/test-my-code created
Go back to the Metrics UI, there is now a Pod showing up named test-my-code in the default namespace. By clicking on this Pod, you can access the logs, you should see the following output:
By clicking on the timeline on the right of the screen, you can access the "Log event document details" panel and notice that many Tekton labels are available out-of-the-box. This is because by default Tekton adds dedicated labels to the Pod created for running the Task, as you can see in the picture below:
You can then filter the logs by the Task name, TaskRun name, and so on
Execute several Tasks and Pipelines and Monitor through a Tekton dashboard
Tekton Pipelines expose some metrics by default in Prometheus format. The metricbeat configuration provided earlier in this post has been initialized with the Prometheus module to automatically gather all these data:
- module: prometheus
period: 10s
hosts:
- http://tekton-pipelines-controller.tekton-pipelines:9090
metrics_path: /metrics
Therefore, your cluster is ready for having dashboards use these data. I have initialized one Tekton Dashboard that you can import using the Kibana Import feature.
This dashboard can be downloaded here.
Once the dashboard is imported, we need to generate data. I have created a script that executes several Tasks and Pipelines from the official repository, you can run this script by doing:
$ git clone https://github.com/mgreau/tekton-pipelines-elastic-o11y.git
$ cd tekton-pipelines-elastic-o11y
# Run Tekton examples from the 0.18.1 git tag in the default namespace
$ ./helpers/add-data.sh 0.18.1 default
Now, open the Tekton Dashboard and watch the number of tasks running grow. You can also see which Tasks take the most time to execute.
I hope that by reading this blog post, you enjoyed seeing how easily Elastic Observability can be used to monitor Tekton CI/CD workflows with minimum configuration.