# Add\_kubernetes\_metadata causes KubeAPIErrorsHigh alert

**URL:** <https://discuss.elastic.co/t/add-kubernetes-metadata-causes-kubeapierrorshigh-alert/245806>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 20, 2020, 5:39pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-causes-kubeapierrorshigh-alert/245806 "2020-08-20T17:39:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yogeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogeek/32/74263_2.png) [@yogeek](https://discuss.elastic.co/u/yogeek)\
**Post date:** [August 20, 2020, 5:39pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-causes-kubeapierrorshigh-alert/245806/1 "2020-08-20T17:39:05Z")

</div>

Hi,

2 days ago, we began to regularly receive the "KubeAPIErrorsHigh" alert on our Kubernetes cluster (definition of the alert [here](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/release-0.4/alerts/kube_apiserver.libsonnet#L75-L89)).

To try to find the cause, we looked at the API server logs and saw the following error messages on `kube-apiserver` pods :

```auto
$ stern kube-apiserver -n kube-system | grep -v "TLS handshake error" | grep -i error

kube-apiserver-ip-10-XX-XX-XX.eu-central-1.compute.internal kube-apiserver I0820 17:13:14.022460 1 log.go:172] http2: server: error reading preface from client 10.XX-XX-XX:55385: read tcp 10.XX-XX-XX:6443->10.XX-XX-XX:55385: read: connection reset by peer
kube-apiserver-ip-10-XX-XX-XX.eu-central-1.compute.internal kube-apiserver E0820 17:14:23.539314 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}
kube-apiserver-ip-10-XX-XX-XX.eu-central-1.compute.internal kube-apiserver E0820 17:14:23.895988 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}

```

After looking at our components one by one to find the culprit, we found that it was our filebeat 7.8.1 daemonset that was reponsible of these errors.  
And then, after many config tests on filebeat config, we concluded that the errors disappeared if we remove the `add_kubernetes_metadata:` field from the `filebeat.yml` config.

And our API server grafana dashboard clearly showed the disappearance of the errors after we removed this option :

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b64d3bb218112fee5100bd4a8febd2bb2f1422d7.png)

Here is the config file we used in our chart deployment :

```auto
---
filebeatConfig:
  filebeat.yml: |
    logging.metrics.enabled: false
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            
            ## Metadata namespace
            - condition.contains:
                kubernetes.namespace: metadata
              config:
                - type: container
                  paths:
                    - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
                  exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines

    processors:
      # Add config here to drop events
      - add_cloud_metadata:
	
      # ******************************** THIS LINE CAUSE API SERVER ERROR
      - add_kubernetes_metadata:

      # Drop all events
      #- drop_event:
      # when.has_fields: ["kubernetes.namespace"]
      # Drop filebeat events
      - drop_event:
          when.equals:
            kubernetes.container.name: "filebeat"
  
    output.logstash:
      hosts: ["logstash.url"]

```

**Environment** :

- K8S version = 1.17.6
- Filebeat versions = 7.8.1 and 7.9 (both tested)
- Filebeat installation : `elastic/filebeat` official char from [https://helm.elastic.co](https://helm.elastic.co)

_Additionnal info : we did not have the problem with our previous 7.5.0 filebeat version. The errors appeared after the 7.5.0 =\> 7.8.1 upgrade._

I was going to open an issue but the message in github mentionned that I had to post a question here before, so here it is 🙂

Do I have to open an issue please ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2020, 7:39pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-causes-kubeapierrorshigh-alert/245806/2 "2020-09-17T19:39:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.