How to enable/hide not necessary k8s pod metrics with metricbeat?

Swathi12 · April 5, 2023, 2:17pm

Hi team,

i successfully getting from my multiple cluster the metrics in Kibana.
But now i see that i don't want ALL metrics from a cluster the pods..

Example: one Cluster has 15 Pods but i need total 8 Pod of them.

Is it possible to hide/enable the pods in metricbeat? How to configure it? Any ideas ?

Thank you

Swathi12 · April 11, 2023, 9:23am

Can someone please help me ?

Andreas_Gkizas · April 11, 2023, 9:54am

Hello @Swathi12 ,

The metricbeat is being deployed as daemonset in all nodes of your kubernetes cluster, thus you collect metrics from all nodes by default.

What you want can be achieved with the use of templates in your manifest. You can define a specific condition based on eg. labels that your 8 pod have in common and you can collect metrics only for them.

Reference under Metricbeat supports templates for modules:

Another relevant example:

templates:
              - condition.and:
                  - equals:
                      kubernetes.namespace: "namespace1"
                  - equals:
                      kubernetes.namespace: "namespace2"

Let me know if that helps

Swathi12 · April 11, 2023, 10:16am

Hi @Andreas_Gkizas

thanks for your quick answer. I need from 16 Pods only 8 Pods.. is that possible ?

Unfortunately i have an error..

Kubernetes is showing this:
Exiting: error loading config file: yaml: line 21: did not find expected key

My .yaml file

metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          unique: true
          templates:
          - condition.and:
                  - equals:
                      kubernetes.deployment.name: "ils"
                  - equals:
                      kubernetes.deployment.name: "ilp"
            - config:

Andreas_Gkizas · April 11, 2023, 1:37pm

Try this one:

 metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          templates:
          - condition.or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "nginx"
            config:

Swathi12 · April 11, 2023, 1:43pm

@Andreas_Gkizas

i don't really understand why to use "kube-system" and "nginx". I need to filter the PODS (from 16 Pods i need 8 Pods )

and i choose the field kubernetes.deployment.name and NOT kubernetes.namespace

getting this error:
Exiting: error loading config file: yaml: line 21: did not find expected key
Exiting: error loading config file: yaml: line 21: did not find expected key
Exiting: error loading config file: yaml: line 21: did not find expected key
Exiting: error loading config file: yaml: line 21: did not find expected key

YAMl:

metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          #unique: true
          templates:
          - condition.or:
                  - equals:
                      kubernetes.deployment.name: "ils"
                  - equals:
                      kubernetes.deployment.name: "ilp"

Andreas_Gkizas · April 11, 2023, 1:59pm

That was an example, you can use whatever conditions you think it matches your scenario.

kubernetes.deployment.name is not available in the kubernetes provider to be used in your conditions. Please have a look here for the available fields:

And here you can find more examples for your conditions:

Swathi12 · April 12, 2023, 2:59pm

@Andreas_Gkizas

i just tried with your given example and its again saying this error: Any idea what is wrong in .YAML fIle?

Exiting: error loading config file: yaml: line 21: did not find expected key

Yaml file starting from Line 17

    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          # In large Kubernetes clusters consider setting unique to false
          # to avoid using the leader election strategy and
          # instead run a dedicated Metricbeat instance using a Deployment in addition to the DaemonSet
          #unique: true
          templates:
              - condition.and:
                  - equals:
                      kubernetes.namespace: "xxxxx"
                  - equals:
                      kubernetes.namespace: "xxxxx"

Andreas_Gkizas · April 13, 2023, 6:53am

I probably suspect an indentation error. From the config you attach can you please move condition on the left and change and with or. (It is a logical or is not it? You need either one or other namespace)

My example:

metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
          - condition.or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "nginx"
            config:
              - module: kubernetes
                hosts: ["kube-state-metrics:8080"]
                period: 10s
                add_metadata: true
                metricsets:
                  - state_node
                  - state_deployment
                  - state_daemonset
                  - state_replicaset
                  - state_pod
                  - state_container
                  - state_job
                  - state_cronjob
                  - state_resourcequota
                  - state_statefulset
                  - state_service
                  - state_persistentvolume
                  - state_persistentvolumeclaim
                  - state_storageclass
              - module: kubernetes
                metricsets:
                  - apiserver
                hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                ssl.certificate_authorities:
                  - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                period: 30s

Swathi12 · April 13, 2023, 7:13am

@Andreas_Gkizas thank you again.

i just set the conditons but why its showing still all pod names? Somehow the filtering is not working correctly

i have total 3 namespaces but instead of 2 its showing 3

Andreas_Gkizas · April 13, 2023, 8:53am

I have been testing in my local cluster as well, the system module is enabled by default (although you dont specify it in the manifest).

So can you check from which metricset the "extra" namespace comes from?
See an example from my local tests with a filter applied and also with metricset.name visible

If you want to disable system metricset just comment:

# metricbeat.config.modules:
    #   # Mounted `metricbeat-daemonset-modules` configmap:
    #   path: ${path.config}/modules.d/*.yml
    #   # Reload module configs as they change:
    #   reload.enabled: false

Also your namespaces are ils/* ?
You can change your filter from equal to contains and be more specific:
eg.

-contains:
     kubernetes.namespace: "xxxxx"

x

Swathi12 · April 13, 2023, 9:26am

@Andreas_Gkizas

i checked with metricset.name and i m getting this

Is the .yml file okai ?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    #metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      #path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      #reload.enabled: false

    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
          - contains:
              kubernetes.namespace: "xxxxxxx"
            config:
              - module: kubernetes
                hosts: ["kube-state-metrics:8080"]
                period: 10s
                add_metadata: true
                metricsets:
                  - state_node
                  - state_deployment
                  - state_daemonset
                  - state_replicaset
                  - state_pod
                  - state_container
                  - state_job
                  - state_cronjob
                  - state_resourcequota
                  - state_statefulset
                  - state_service
                  - state_persistentvolume
                  - state_persistentvolumeclaim
                  - state_storageclass
              - module: kubernetes
                metricsets:
                  #- apiserver
               # hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                ssl.certificate_authorities:
                  - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                period: 30s

system · May 11, 2023, 9:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kubernetes autodiscover template not working Beats metricbeat	1	308	October 3, 2019
Can i get the pod labels in the metricsets Beats metricbeat	5	702	December 3, 2018
Metricbeat with namespace secrets in Kubernetes Beats metricbeat	4	814	May 22, 2019
Metricbeat on kubernetes - Desired pods and Available pods are wrong Beats metricbeat	7	1529	August 2, 2018
Metric beat kubernetes example not showing pod metrics Beats metricbeat	2	998	August 17, 2019

How to enable/hide not necessary k8s pod metrics with metricbeat?

Related Topics