# How to enable/hide not necessary k8s pod metrics with metricbeat?

**URL:** <https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437>\
**Category:** Kibana\
**Created:** [April 5, 2023, 2:17pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437 "2023-04-05T14:17:52Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 5, 2023, 2:17pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/1 "2023-04-05T14:17:52Z")

</div>

Hi team,

i successfully getting from my multiple cluster the metrics in Kibana.  
But now i see that i don't want ALL metrics from a cluster the pods..

Example: one Cluster has 15 Pods but i need total 8 Pod of them.

Is it possible to hide/enable the pods in metricbeat? How to configure it? Any ideas ?

Thank you

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 11, 2023, 9:23am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/2 "2023-04-11T09:23:51Z")

</div>

Can someone please help me ?

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [April 11, 2023, 9:54am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/3 "2023-04-11T09:54:22Z")

</div>

Hello @Swathi12 ,

The metricbeat is being deployed as daemonset in all nodes of your kubernetes cluster, thus you collect metrics from all nodes by default.

What you want can be achieved with the use of templates in your manifest. You can define a specific condition based on eg. labels that your 8 pod have in common and you can collect metrics only for them.

Reference under `Metricbeat supports templates for modules:`

> **[Autodiscover | Metricbeat Reference \[8.7\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html)**

Another relevant example:

```yaml
templates:
              - condition.and:
                  - equals:
                      kubernetes.namespace: "namespace1"
                  - equals:
                      kubernetes.namespace: "namespace2"

```

Let me know if that helps

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 11, 2023, 10:16am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/4 "2023-04-11T10:16:33Z")

</div>

Hi @Andreas_Gkizas

thanks for your quick answer. I need from 16 Pods only 8 Pods.. is that possible ?

Unfortunately i have an error..

Kubernetes is showing this:  
Exiting: error loading config file: yaml: line 21: did not find expected key

My .yaml file

```auto
metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          unique: true
          templates:
          - condition.and:
                  - equals:
                      kubernetes.deployment.name: "ils"
                  - equals:
                      kubernetes.deployment.name: "ilp"
            - config:

```

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [April 11, 2023, 1:37pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/5 "2023-04-11T13:37:24Z")

</div>

Try this one:

```yaml
 metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          templates:
          - condition.or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "nginx"
            config:

```

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 11, 2023, 1:43pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/6 "2023-04-11T13:43:05Z")

</div>

@Andreas_Gkizas

i don't really understand why to use "kube-system" and "nginx". **I need to filter the PODS (from 16 Pods i need 8 Pods )**

and i choose the field **kubernetes.deployment.name** and **NOT kubernetes.namespace**

getting this error:  
Exiting: error loading config file: yaml: line 21: did not find expected key  
Exiting: error loading config file: yaml: line 21: did not find expected key  
Exiting: error loading config file: yaml: line 21: did not find expected key  
Exiting: error loading config file: yaml: line 21: did not find expected key

YAMl:

```auto
metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          #unique: true
          templates:
          - condition.or:
                  - equals:
                      kubernetes.deployment.name: "ils"
                  - equals:
                      kubernetes.deployment.name: "ilp"
```

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [April 11, 2023, 1:59pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/7 "2023-04-11T13:59:59Z")

</div>

That was an example, you can use whatever conditions you think it matches your scenario.

kubernetes.deployment.name is not available in the kubernetes provider to be used in your conditions. Please have a look here for the available fields:

> **[Autodiscover | Metricbeat Reference \[8.7\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html#_generic_fields)**

And here you can find more examples for your conditions:

> **[Define processors | Metricbeat Reference \[8.7\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/defining-processors.html#conditions)**

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 12, 2023, 2:59pm UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/8 "2023-04-12T14:59:33Z")

</div>

@Andreas_Gkizas

i just tried with your given example and its again saying this error: Any idea what is wrong in .YAML fIle?

Exiting: error loading config file: yaml: line 21: did not find expected key

Yaml file starting from Line 17

```auto
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          #unique: true
          templates:
              - condition.and:
                  - equals:
                      kubernetes.namespace: "xxxxx"
                  - equals:
                      kubernetes.namespace: "xxxxx"

```

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [April 13, 2023, 6:53am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/9 "2023-04-13T06:53:03Z")

</div>

I probably suspect an indentation error. From the config you attach can you please move condition on the left and change and with or. (It is a logical or is not it? You need either one or other namespace)

My example:

```auto
metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
          - condition.or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "nginx"
            config:
              - module: kubernetes
                hosts: ["kube-state-metrics:8080"]
                period: 10s
                add_metadata: true
                metricsets:
                  - state_node
                  - state_deployment
                  - state_daemonset
                  - state_replicaset
                  - state_pod
                  - state_container
                  - state_job
                  - state_cronjob
                  - state_resourcequota
                  - state_statefulset
                  - state_service
                  - state_persistentvolume
                  - state_persistentvolumeclaim
                  - state_storageclass
              - module: kubernetes
                metricsets:
                  - apiserver
                hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                ssl.certificate_authorities:
                  - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                period: 30s

```

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 13, 2023, 7:13am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/10 "2023-04-13T07:13:37Z")

</div>

@Andreas_Gkizas thank you again.

i just set the conditons but why its showing still all pod names? Somehow the filtering is not working correctly

i have total 3 namespaces but instead of 2 its showing 3

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/3/73d3fd70649cd0a05dc9f208625354d7e6a354cf.png)

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [April 13, 2023, 8:53am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/11 "2023-04-13T08:53:35Z")

</div>

I have been testing in my local cluster as well, the system module is enabled by default (although you dont specify it in the manifest).

So can you check from which metricset the "extra" namespace comes from?  
See an example from my local tests with a filter applied and also with metricset.name visible

 ![Screenshot 2023-04-13 at 11.47.43 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c4981d0ed7e90fe366340124b55ca0e2b8d2d72.png)

If you want to disable system metricset just comment:

```yaml
# metricbeat.config.modules:
    # # Mounted `metricbeat-daemonset-modules` configmap:
    # path: ${path.config}/modules.d/*.yml
    # # Reload module configs as they change:
    # reload.enabled: false

```

Also your namespaces are ils/\* ?  
You can change your filter from equal to contains and be more specific:  
eg.

```yaml
-contains:
     kubernetes.namespace: "xxxxx"

```

x

---

<div class="post-metadata">

**Author:** ![Swathi12](https://avatars.discourse-cdn.com/v4/letter/s/a5b964/32.png) [@Swathi12](https://discuss.elastic.co/u/Swathi12)\
**Post date:** [April 13, 2023, 9:26am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/12 "2023-04-13T09:26:36Z")

</div>

@Andreas_Gkizas

i checked with metricset.name and i m getting this

![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/119762d73a357e5a00a51750aeaeeb147af11209.png)

* * *

Is the .yml file okai ?

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    #metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      #path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      #reload.enabled: false

    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
          - contains:
              kubernetes.namespace: "xxxxxxx"
            config:
              - module: kubernetes
                hosts: ["kube-state-metrics:8080"]
                period: 10s
                add_metadata: true
                metricsets:
                  - state_node
                  - state_deployment
                  - state_daemonset
                  - state_replicaset
                  - state_pod
                  - state_container
                  - state_job
                  - state_cronjob
                  - state_resourcequota
                  - state_statefulset
                  - state_service
                  - state_persistentvolume
                  - state_persistentvolumeclaim
                  - state_storageclass
              - module: kubernetes
                metricsets:
                  #- apiserver
               # hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                ssl.certificate_authorities:
                  - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                period: 30s

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 11, 2023, 9:26am UTC](https://discuss.elastic.co/t/how-to-enable-hide-not-necessary-k8s-pod-metrics-with-metricbeat/329437/13 "2023-05-11T09:26:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.