Add_kubernetes_metadata causes KubeAPIErrorsHigh alert


2 days ago, we began to regularly receive the "KubeAPIErrorsHigh" alert on our Kubernetes cluster (definition of the alert here).

To try to find the cause, we looked at the API server logs and saw the following error messages on kube-apiserver pods :

$ stern kube-apiserver -n kube-system | grep -v "TLS handshake error" | grep -i error kube-apiserver I0820 17:13:14.022460       1 log.go:172] http2: server: error reading preface from client 10.XX-XX-XX:55385: read tcp 10.XX-XX-XX:6443->10.XX-XX-XX:55385: read: connection reset by peer kube-apiserver E0820 17:14:23.539314       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} kube-apiserver E0820 17:14:23.895988       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}

After looking at our components one by one to find the culprit, we found that it was our filebeat 7.8.1 daemonset that was reponsible of these errors.
And then, after many config tests on filebeat config, we concluded that the errors disappeared if we remove the add_kubernetes_metadata: field from the filebeat.yml config.

And our API server grafana dashboard clearly showed the disappearance of the errors after we removed this option :

Here is the config file we used in our chart deployment :

  filebeat.yml: |
    logging.metrics.enabled: false
        - type: kubernetes
            ## Metadata namespace
            - condition.contains:
                kubernetes.namespace: metadata
                - type: container
                    - /var/lib/docker/containers/${}/*.log
                  exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

      # Add config here to drop events
      - add_cloud_metadata:
      # ******************************** THIS LINE CAUSE API SERVER ERROR
      - add_kubernetes_metadata:

      # Drop all events
      #- drop_event:
      #    when.has_fields: ["kubernetes.namespace"]
      # Drop filebeat events
      - drop_event:
      hosts: ["logstash.url"]

Environment :

  • K8S version = 1.17.6
  • Filebeat versions = 7.8.1 and 7.9 (both tested)
  • Filebeat installation : elastic/filebeat official char from

Additionnal info : we did not have the problem with our previous 7.5.0 filebeat version. The errors appeared after the 7.5.0 => 7.8.1 upgrade.

I was going to open an issue but the message in github mentionned that I had to post a question here before, so here it is :slight_smile:

Do I have to open an issue please ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.