Metricbeat not processing kubernetes labels

Elastic Stack Beats
1

Hello,

I'm running Metrict beat 6.3.1 on a Kubernetes clusters on AWS. It works fine, except for the fact that the kubernetes.labels field is missing. The metricbeat index has 1330 fields, but searching for "label" returns only kubernetes.replicaset.replicas.labeled.

I'm deploying with kubectl using the config from here: https://github.com/elastic/beats/blob/master/deploy/kubernetes

Below is my metricbeat-config ConfigMap. I have added the "add_kubernetes_metadata" processor which was supposed to get all labels, but does not seem to work.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          in_cluster: true
          
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_URL}']
      pipeline: kube
---

I've also tried adding include_labels/include_annotations under metricbeat-daemonset-modules ConfigMap as per https://github.com/elastic/beats/pull/7470 but to no avail :confused:

Metricbeat DaemonSets are not logging any errors either, so I'm completely lost Any idea what could be the issue or where else could I look for information on this?

Edit: by the way we're running filebeat on the same cluster and it can get labels just fine.

Cheers

1 Like
2

Hi @fbcbarbosa and welcome :slight_smile:

Indeed labels should appear, on what events you are missing the labels? could you share the modules configuration?

3

Hi @jsoriano, thanks!

On all of them. Although the ones that are relevenat to me are container and pod.

I have the following modules configured:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      hosts: ["localhost:10255"]

and

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  # This module requires `kube-state-metrics` up and running under `kube-system` namespace
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        - event
      period: 10s
      hosts: ["kube-state-metrics:8080"]

Also, kube-state-metrics is running.

4

One important difference between the filebeat and metricbeat deployments that are available in the repository is that metricbeat is run in the host network (hostNetwork: true). With host network processors and modules that need connectivity with the API server may require additional configuration. If this is the problem in your case you may be seeing errors when starting metricbeat like kubernetes: Querying for pod failed with error....

The additional configuration that may be needed is the node host, it can be added with the host setting:

      - add_kubernetes_metadata:
          host: ${NODE_NAME}

NODE_NAME has to be injected in the metricbeat spec by adding it to the list of environment variables like this:

        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

By default it should be able to add labels to pods that match by ip and port (notice that this implies that your containers need to have ports defined). If you need to set additional rules to match objects, take a look to indexers and matchers configuration.

5

Thanks @jsoriano, but unfortunately that did not work :confused:

I was indeed getting an error when starting up metricbeat like you said:

ERROR	kubernetes/util.go:73	kubernetes: Querying for pod failed with error: %!(EXTRA string=kubernetes api: Failure 404 pods "ip-10-11-13-123" not found)

I've added the NODE name variable as you said and the error is gone. Testing shows that the variable is loaded correctly:

$ kubectl exec -n kube-system -it metricbeat-t22pp -- env | grep NODE
NODE_NAME=ip-10-11-12-11.ec2.internal

And my containers do have ports defined. Also, I guess this shouldn't be the issue, as I am getting a ton of metrics, just not k8s labels/annotations (cloud metadata works fine).

Isn't there some configuration that works out of the box for a kubernetes cluster? Maybe one that does not run on hostNetwork? For filebeat I just ran the helm chart and it worked like a charm (no official helm chart is available for metricbeat though).

EDIT: I'm not sure hostNetwork would be the way to go though. According to add_kubernetes_metadata, host is only required when in_cluster is set to false, which is not the case, as I am running metricbeat as part of the cluster. And if I set in_cluster to false, metricbeat breaks anyway:

2018-08-21T15:02:14.681Z	ERROR	instance/beat.go:691	Exiting: error initializing publisher: error initializing processors: `kube_config` path can't be empty when in_cluster is set to false
Exiting: error initializing publisher: error initializing processors: `kube_config` path can't be empty when in_cluster is set to false
6

Hi @fbcbarbosa,

Isn't there some configuration that works out of the box for a kubernetes cluster? Maybe one that does not run on hostNetwork? For filebeat I just ran the helm chart and it worked like a charm (no official helm chart is available for metricbeat though).

Host network in metricbeat is recommended to collect also network metrics of the host and this is not needed in filebeat. In principle metricbeat can be run without host network mode if you are not interested on metrics of the host node.

EDIT: I'm not sure hostNetwork would be the way to go though. According to add_kubernetes_metadata, host is only required when in_cluster is set to false , which is not the case, as I am running metricbeat as part of the cluster. And if I set in_cluster to false , metricbeat breaks anyway:

Yes, the documentation is not very clear for some of these settings, we have started to do some improvements there (#8036, #8029).

Regarding add_kubernetes_metadata, it is more intended to be used on events of non-kubernetes metricsets, for example to add the pod metadata to the metrics collected by other modules like nginx. This is a bit tricky when used with the kubernetes module, the indexers and matchers have to be modified and when it works it can unexpectedly overwrite other metrics under kubernetes namespace.

In 6.4 all the kubernetes metricsets will also collect labels when possible out of the box, probably the best option in your case is to wait for this release, it is expected to happen on the following days.

7

Great, I'll wait to try out the new release then.

Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.