Dec 16th, 2020: [FR] Monitorer les tâches et pipelines Tekton avec Elastic Observability

English Version

Savez-vous que Elastic a delivre 21 releases en 2020?

Chaque fois qu'une version est delivree, ce sont 500+ artifacts publiés dans multiplies emplacements publics (bucket, registres Docker, Maven Central, Rubygems...) et evidement disponbile sur Elastic Cloud dans le même temps. Ce process complexe est devenu

This complex process became a non-event thanks to our Unified Release workflow based on Tekton Tasks and Pipelines and monitored with Elastic Observability.

This blog post shows how to run your first Tekton Task, and then how to install and use the Elastic Observability Solution to monitor many Tasks and Pipelines deployed within a cluster. All you need to have is a Kubernetes cluster available locally and 10 minutes of your time. Elastic Cloud on Kubernetes (ECK) is used to set up the Elastic components.

Note: the examples are available in the tekton-pipelines-elastic-o11y GitHub repository

What’s Tekton Pipelines?

Tekton is an open-source framework for creating CI/CD systems. Tekton Pipelines are the building blocks of Tekton CI/CD workflows. It’s a Kubernetes extension that defines a set of Kubernetes Custom resources (CRDs) from which the workflows are assembled.

The main entities are the following:

  • Step: the most basic building block, this is a Kubernetes container specification
  • Task: a collection of steps that run in sequential order in the same Kubernetes node
  • TaskRun: to instantiate and execute a Task on a Kubernetes cluster
  • Pipeline: combine Tasks together, that run sequentially or concurrently
  • PipelineRun: to instantiate and execute a Pipeline on a Kubernetes cluster

Checkout the official website at tektoncd.dev/docs to have more information.

Notes: To keep it simple, this blog post focuses on deploying and running Tasks and TaskRuns, and it does not introduce Pipeline and PipelineRuns resources.

Get your first Tekton Task running!

Now that you know what Tekton Pipelines is, let’s see how to execute a Task. The first steps are to install Tekton Pipelines to your local cluster and then install the Tekton CLI to interact with it.

Install Tekton Pipelines and Tekton CLI (tkn)

Installing Tekton Pipelines is done through a single command line

$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml

You can refer to the official documentation to set up the CLI for your specific environment. For example, for macOS users, tkn is available via brew:

$ brew install tektoncd-cli

Make sure both components are successfully installed by running:

$ tkn version
Client version: 0.14.0
Pipeline version: v0.18.1

Execute your first Task

The goal of this first Task is to checkout the source code from a git repository to your Kubernetes cluster, so you are able to run the tests from this project afterwards.

The commands below install the git-clone Task into the cluster from the Tekton Catalog and create a PersistentVolumeClaim (PVC) to store the source code and share it across multiple Tasks.

# Install the git-clone Task from Tekton Catalog
$ tkn hub get task git-clone --version 0.2 |  kubectl apply -f -
task.tekton.dev/git-clone created
 
# Create a PVC to share data between tasks
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: go-source
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
EOF
persistentvolumeclaim/go-source created

Everything is in place to execute a Task through the deployment of a first TaskRun that is going to invoke the Task:

# Execute a Task to clone the project
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

Great, the TaskRun is deployed and you can check the result. You can also see the logs through the CLI commands:

$ tkn taskruns list
NAME                                                  STARTED        DURATION     STATUS
clone-my-code                                         1 minute ago   9 seconds    Succeeded

$ tkn taskruns logs clone-my-code
...
[clone] + /ko-app/git-init -url https://github.com/elastic/go-licenser.git -revision v0.3.1 -refspec  -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[clone] {"level":"info","ts":1607989263.4070501,"caller":"git/git.go:165","msg":"Successfully cloned https://github.com/elastic/go-licenser.git @ 857b4969bc2f753ffb9eb3a885d01a59a9f22cdb (grafted, HEAD) in path /workspace/output/"}
[clone] ...

So far, you have executed a first Task, and checked the output using the CLI.

Next, before executing the Task for running the tests of this project, let’s install the Elastic Observability Solution.

Setup Elastic Observability with Elastic Cloud on Kubernetes

Elastic Cloud on Kubernetes (ECK) is the official operator for provisioning Elastic Stack deployments in Kubernetes. This article is not going to explain how it works. To know more about it, checkout the dedicated blog post Elastic Stack Monitoring with Elastic Cloud on Kubernetes

Observability with the Elastic Stack allows unifying all your logs, metrics, and application trace information in one place. Take a look at the Observability with the Elastic Stack blog post to know more about it.

Install ECK

Installing ECK is done through a single command line:

$ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml 

Install Elastic Observability

Installing the Elastic Observability Solution is done in two steps here. First, Elasticsearch and Kibana are deployed, then Metricbeat and Filebeat.

# Deploy Elasticsearch and Kibana
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-es-kb.yaml

# Deploy Metricbeat and Filebeat
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-filebeat-metricbeat.yaml 

Check that everything is up and running (wait until filebeat and metricbeat are running for more than 3 minutes):

# Check Elastic pods
$ kubectl get pods -n elastic-system
NAME                                    READY   STATUS    RESTARTS   AGE
elastic-operator-0                      1/1     Running   1          5m
elasticsearch-monitoring-es-default-0   1/1     Running   1          5m
filebeat-beat-filebeat-b6vlt            1/1     Running   7          5m
kibana-monitoring-kb-599698987-7cjqq    1/1     Running   1          5m
metricbeat-beat-metricbeat-fsz5p        1/1     Running   7          5m

Get the password for the elastic user needed to access the UI:

# Password for the elastic user
$ echo $(kubectl get secret -n elastic-system elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
XXXXXXXXXXXXXX

Make Kibana available on port 5601:

$ kubectl port-forward -n elastic-system svc/kibana-monitoring-kb-http 5601

Now, you can access Elastic Observability (use the above credentials):

Note: the intent of this tutorial is to provide a development environment. To install a valid certificate, please refer to the official Elastic documentation.

Go to the Metrics UI, in the Inventory view, and click on Show the Kubernetes Pods. Then select Grouped by Namespace. You should see the Pods running in the elastic-system namespace and the ones running in the tekton-pipelines namespace.

Let's execute more Tasks and see how it goes.

Monitor Tasks and Pipelines with Elastic Observability

Run the tests Task and check the logs via the Logs UI

In this step, we are going to execute a Task and automatically see the logs via the UI.

# Install the generic golang-test Task from the catalog
$ tkn hub get task golang-test --version 0.1 |  kubectl apply -f -
task.tekton.dev/golang-test created
 
# Execute a Task to run the tests
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: test-my-code
spec:
  taskRef:
    name: golang-test
  workspaces:
  - name: source
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: package
    value: github.com/elastic/go-licenser
  - name: packages
    value: ./...
  - name: flags
    value: -timeout 10s -p 4 -race -cover
EOF
taskrun.tekton.dev/test-my-code created

Go back to the Metrics UI, there is now a Pod showing up named test-my-code in the default namespace. By clicking on this Pod, you can access the logs, you should see the following output:

By clicking on the timeline on the right of the screen, you can access the "Log event document details" panel and notice that many Tekton labels are available out-of-the-box. This is because by default Tekton adds dedicated labels to the Pod created for running the Task, as you can see in the picture below:

You can then filter the logs by the Task name, TaskRun name, and so on

Execute several Tasks and Pipelines and Monitor through a Tekton dashboard

Tekton Pipelines expose some metrics by default in Prometheus format. The metricbeat configuration provided earlier in this post has been initialized with the Prometheus module to automatically gather all these data:

     - module: prometheus
        period: 10s
        hosts:
        - http://tekton-pipelines-controller.tekton-pipelines:9090
        metrics_path: /metrics

Therefore, your cluster is ready for having dashboards use these data. I have initialized one Tekton Dashboard that you can import using the Kibana Import feature.
This dashboard can be downloaded here.

Once the dashboard is imported, we need to generate data. I have created a script that executes several Tasks and Pipelines from the official repository, you can run this script by doing:

$ git clone https://github.com/mgreau/tekton-pipelines-elastic-o11y.git
$ cd tekton-pipelines-elastic-o11y
# Run Tekton examples from the 0.18.1 git tag in the default namespace
$ ./helpers/add-data.sh 0.18.1 default

Now, open the Tekton Dashboard and watch the number of tasks running grow. You can also see which Tasks take the most time to execute.

I hope that by reading this blog post, you enjoyed seeing how easily Elastic Observability can be used to monitor Tekton CI/CD workflows with minimum configuration.

1 Like