# Dec 16th, 2020: \[FR\] Monitorer les tâches et pipelines Tekton avec Elastic Observability

**URL:** <https://discuss.elastic.co/t/dec-16th-2020-fr-monitorer-les-taches-et-pipelines-tekton-avec-elastic-observability/258847>\
**Category:** Advent Calendar\
**Created:** [January 6, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-16th-2020-fr-monitorer-les-taches-et-pipelines-tekton-avec-elastic-observability/258847 "2021-01-06T08:00:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgreau](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mgreau/32/22607_2.png) [@mgreau](https://discuss.elastic.co/u/mgreau)\
**Post date:** [January 6, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-16th-2020-fr-monitorer-les-taches-et-pipelines-tekton-avec-elastic-observability/258847/1 "2021-01-06T08:00:04Z")

</div>

[English Version](https://discuss.elastic.co/t/dec-16th-2020-en-monitoring-tekton-tasks-and-pipelines-with-elastic-observability/257567)

Savez-vous que Elastic a delivre 21 releases en 2020?

Chaque fois qu'une version est delivree, ce sont 500+ artifacts publiés dans multiplies emplacements publics (bucket, registres Docker, Maven Central, Rubygems...) et evidement disponbile sur Elastic Cloud dans le même temps. Ce process complexe est devenu

This complex process became a non-event thanks to our Unified Release workflow based on **Tekton Tasks and Pipelines** and monitored with **[Elastic Observability](https://www.elastic.co/observability)**.

This blog post shows how to run your first Tekton Task, and then how to install and use the Elastic Observability Solution to monitor many Tasks and Pipelines deployed within a cluster. All you need to have is a Kubernetes cluster available locally and **10 minutes of your time**. [**Elastic Cloud on Kubernetes**](https://www.elastic.co/elastic-cloud-kubernetes) (ECK) is used to set up the Elastic components.

_Note: the examples are available in the [tekton-pipelines-elastic-o11y](https://github.com/mgreau/tekton-pipelines-elastic-o11y)GitHub repository_

## What’s Tekton Pipelines?

Tekton is an open-source framework for creating CI/CD systems. [Tekton Pipelines](https://github.com/tektoncd/pipeline) are **the building blocks of Tekton CI/CD workflows**. It’s a Kubernetes extension that defines a set of [Kubernetes Custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs) from which the workflows are assembled.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/a/1/a139456258ef730640fb32b0893f879b489e3711.png)

The main entities are the following:

- `Step`: the most basic building block, this is a Kubernetes container specification
- `Task`: a collection of steps that run in sequential order in the same Kubernetes node
- `TaskRun`: to instantiate and execute a Task on a Kubernetes cluster
- `Pipeline`: combine Tasks together, that run sequentially or concurrently
- `PipelineRun`: to instantiate and execute a Pipeline on a Kubernetes cluster

Checkout the official website at [tektoncd.dev/docs](https://tekton.dev/docs/) to have more information.

_Notes: To keep it simple, this blog post focuses on deploying and running Tasks and TaskRuns, and it does not introduce Pipeline and PipelineRuns resources._

## Get your first Tekton Task running!

Now that you know what Tekton Pipelines is, let’s see how to execute a `Task`. The first steps are to install Tekton Pipelines to your local cluster and then install the Tekton CLI to interact with it.

### Install Tekton Pipelines and Tekton CLI (tkn)

Installing Tekton Pipelines is done through a single command line

```auto
$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml

```

You can refer to the official documentation to [set up the CLI for your specific environment](https://tekton.dev/docs/getting-started/#set-up-the-cli). For example, for macOS users, `tkn` is available via brew:

```auto
$ brew install tektoncd-cli

```

Make sure both components are successfully installed by running:

```auto
$ tkn version
Client version: 0.14.0
Pipeline version: v0.18.1

```

### Execute your first Task

The goal of this first `Task` is to checkout the source code from a git repository to your Kubernetes cluster, so you are able to run the tests from this project afterwards.

The commands below install the [git-clone Task](https://github.com/tektoncd/catalog/tree/master/task/git-clone/0.2) into the cluster from the Tekton Catalog and create a [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVC) to store the source code and share it across multiple Tasks.

```auto
# Install the git-clone Task from Tekton Catalog
$ tkn hub get task git-clone --version 0.2 | kubectl apply -f -
task.tekton.dev/git-clone created
 
# Create a PVC to share data between tasks
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: go-source
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
EOF
persistentvolumeclaim/go-source created

```

Everything is in place to execute a `Task` through the deployment of a first `TaskRun` that is going to invoke the `Task`:

```auto
# Execute a Task to clone the project
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

```

Great, the `TaskRun` is deployed and you can check the result. You can also see the logs through the CLI commands:

```auto
$ tkn taskruns list
NAME STARTED DURATION STATUS
clone-my-code 1 minute ago 9 seconds Succeeded

$ tkn taskruns logs clone-my-code
...
[clone] + /ko-app/git-init -url https://github.com/elastic/go-licenser.git -revision v0.3.1 -refspec -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[clone] {"level":"info","ts":1607989263.4070501,"caller":"git/git.go:165","msg":"Successfully cloned https://github.com/elastic/go-licenser.git @ 857b4969bc2f753ffb9eb3a885d01a59a9f22cdb (grafted, HEAD) in path /workspace/output/"}
[clone] ...

```

So far, you have executed a first `Task`, and checked the output using the CLI.

Next, before executing the `Task` for running the tests of this project, let’s install the **Elastic Observability Solution**.

## Setup Elastic Observability with Elastic Cloud on Kubernetes

Elastic Cloud on Kubernetes (ECK) is the official operator for provisioning Elastic Stack deployments in Kubernetes. This article is not going to explain how it works. To know more about it, checkout the dedicated blog post [Elastic Stack Monitoring with Elastic Cloud on Kubernetes](https://www.elastic.co/blog/elastic-stack-monitoring-with-elastic-cloud-on-kubernetes)

Observability with the Elastic Stack allows unifying all your logs, metrics, and application trace information in one place. Take a look at the [Observability with the Elastic Stack](https://www.elastic.co/blog/observability-with-the-elastic-stack) blog post to know more about it.

### Install ECK

Installing ECK is done through a single command line:

```auto
$ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml 

```

### Install Elastic Observability

Installing the Elastic Observability Solution is done in two steps here. First, Elasticsearch and Kibana are deployed, then Metricbeat and Filebeat.

```auto
# Deploy Elasticsearch and Kibana
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-es-kb.yaml

# Deploy Metricbeat and Filebeat
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-filebeat-metricbeat.yaml 

```

Check that everything is up and running (wait until filebeat and metricbeat are running for more than 3 minutes):

```auto
# Check Elastic pods
$ kubectl get pods -n elastic-system
NAME READY STATUS RESTARTS AGE
elastic-operator-0 1/1 Running 1 5m
elasticsearch-monitoring-es-default-0 1/1 Running 1 5m
filebeat-beat-filebeat-b6vlt 1/1 Running 7 5m
kibana-monitoring-kb-599698987-7cjqq 1/1 Running 1 5m
metricbeat-beat-metricbeat-fsz5p 1/1 Running 7 5m

```

Get the password for the elastic user needed to access the UI:

```auto
# Password for the elastic user
$ echo $(kubectl get secret -n elastic-system elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
XXXXXXXXXXXXXX

```

Make Kibana available on port 5601:

```auto
$ kubectl port-forward -n elastic-system svc/kibana-monitoring-kb-http 5601

```

Now, you can access Elastic Observability (use the above credentials):

- [https://localhost:5601/app/observability/overview](https://localhost:5601/app/observability/overview)

_Note: the intent of this tutorial is to provide a development environment. To install a valid certificate, please refer to the official Elastic documentation._

Go to the Metrics UI, in the Inventory view, and click on _Show the Kubernetes Pods_. Then select _Grouped by Namespace_. You should see the Pods running in the `elastic-system` namespace and the ones running in the `tekton-pipelines` namespace.

 ![elastic-o11-tekton-1](https://us1.discourse-cdn.com/elastic/original/3X/a/e/ae4a44a6f6efa72d46464301caaeef7d269c78c9.png)

Let's execute more Tasks and see how it goes.

## Monitor Tasks and Pipelines with Elastic Observability

### Run the tests Task and check the logs via the Logs UI

In this step, we are going to execute a Task and automatically see the logs via the UI.

```auto
# Install the generic golang-test Task from the catalog
$ tkn hub get task golang-test --version 0.1 | kubectl apply -f -
task.tekton.dev/golang-test created
 
# Execute a Task to run the tests
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: test-my-code
spec:
  taskRef:
    name: golang-test
  workspaces:
  - name: source
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: package
    value: github.com/elastic/go-licenser
  - name: packages
    value: ./...
  - name: flags
    value: -timeout 10s -p 4 -race -cover
EOF
taskrun.tekton.dev/test-my-code created

```

Go back to the Metrics UI, there is now a Pod showing up named `test-my-code` in the `default` namespace. By clicking on this Pod, you can access the logs, you should see the following output:

 ![elastic-o11-tekton-2](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8cee9bc35ac405dc0c84f6f89cce68d232b2aecd.png)

 ![elastic-o11-tekton-3](https://us1.discourse-cdn.com/elastic/original/3X/6/4/6454a91d2d7b086b694b9d9bd9151230695f9d40.png)

By clicking on the timeline on the right of the screen, you can access the "Log event document details" panel and notice that many Tekton labels are available out-of-the-box. This is because by default Tekton adds dedicated labels to the Pod created for running the `Task`, as you can see in the picture below:

 ![elastic-o11-tekton-4](https://us1.discourse-cdn.com/elastic/original/3X/3/9/39d38016bcfbccec69294c618e55a3642956a6e8.png)

You can then filter the logs by the `Task` name, `TaskRun` name, and so on

 ![elastic-o11-tekton-5](https://us1.discourse-cdn.com/elastic/original/3X/2/9/2914089744bc775f30169d0f22b1588cfb5e8af2.png)

### Execute several Tasks and Pipelines and Monitor through a Tekton dashboard

Tekton Pipelines expose [some metrics by default](https://github.com/tektoncd/pipeline/blob/v0.18.1/docs/metrics.md) in Prometheus format. The [metricbeat configuration provided earlier](https://github.com/mgreau/tekton-pipelines-elastic-o11y/blob/5fff77ed99af2daa3ca362afcd3b9c58785905f2/config/eck/monitoring-filebeat-metricbeat.yaml#L168-L172) in this post has been initialized with the **Prometheus module** to automatically gather all these data:

```auto
     - module: prometheus
        period: 10s
        hosts:
        - http://tekton-pipelines-controller.tekton-pipelines:9090
        metrics_path: /metrics

```

Therefore, your cluster is ready for having dashboards use these data. I have initialized one **Tekton Dashboard** that you can **import** using the [Kibana Import feature](https://release-stats.elastic.co/app/management/kibana/objects).  
This dashboard can be downloaded [here](https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-o11y/master/helpers/tekton-dashboard.ndjson).

Once the dashboard is imported, we need to generate data. I have created a script that executes several [Tasks and Pipelines from the official repository](https://github.com/tektoncd/pipeline/tree/v0.18.1/examples/v1beta1), you can run this script by doing:

```auto
$ git clone https://github.com/mgreau/tekton-pipelines-elastic-o11y.git
$ cd tekton-pipelines-elastic-o11y
# Run Tekton examples from the 0.18.1 git tag in the default namespace
$ ./helpers/add-data.sh 0.18.1 default

```

Now, open the **Tekton Dashboard** and watch the number of tasks running grow. You can also see which Tasks take the most time to execute.

 ![elastic-o11-tekton-6bis](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76caff6cea5e6c389b6c0ff37344fdfad354e74b.png)

 ![elastic-o11-tekton-6](https://us1.discourse-cdn.com/elastic/original/3X/7/3/736ca13bc9e8275478db1dc4959a3c3c0e213ed4.png)

I hope that by reading this blog post, you enjoyed seeing how easily [Elastic Observability](https://www.elastic.co/observability) can be used to monitor [Tekton CI/CD workflows](https://tekton.dev/docs/concepts/) with minimum configuration.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 3, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-16th-2020-fr-monitorer-les-taches-et-pipelines-tekton-avec-elastic-observability/258847/2 "2021-02-03T08:00:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.