# Filebeat errors while connecting to kubernetes API on start filebeat POD **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 25, 2019, 3:43am UTC](https://discuss.elastic.co/t/filebeat-errors-while-connecting-to-kubernetes-api-on-start-filebeat-pod/192158 "2019-07-25T03:43:26Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![alexMX](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@alexMX](https://discuss.elastic.co/u/alexMX)\ **Post date:** [July 25, 2019, 3:43am UTC](https://discuss.elastic.co/t/filebeat-errors-while-connecting-to-kubernetes-api-on-start-filebeat-pod/192158/1 "2019-07-25T03:43:27Z")
Hello! On our k8s infrastructure, we are face to the connection issue while starting the filebeat POD. Affected filebeat versions are 6.4.0, 7.0. Log record is: > ERROR kubernetes/kubernetes.go:127 Error starting kubernetes autodiscover provider: performing request: Get [https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:](https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:) dial tcp 10.96.0.1:443: connect: connection refused The k8s manifest configuration is: > **filebeat-daemonset.yml** > > ``` > --- > apiVersion: v1 > kind: ConfigMap > metadata: > name: filebeat-config-json > labels: > k8s-app: filebeat > data: > filebeat.yml: |- > filebeat.registry_file: "/opt/filebeat/registry" > logging.to_stderr: true > filebeat.autodiscover: > providers: > - type: kubernetes > include_annotations: ["logging"] > templates: > - condition: > equals: > kubernetes.annotations.logging: "plain" > config: > - type: docker > containers.ids: > - "${data.kubernetes.container.id}" > processors: > - add_cloud_metadata: ~ > processors: > - drop_fields: > fields: ["host"] > output.logstash: > hosts: ["logstash-host:9200"] > logging: > to_files: true > files: > path: "/opt/filebeat/logs" > name: filebeat.log > rotateeverybytes: 10485760 # = 10MB > keepfiles: 7 > level: info > > --- > apiVersion: apps/v1 > kind: DaemonSet > metadata: > name: filebeat-daemon > labels: > k8s-app: filebeat > spec: > selector: > matchLabels: > name: filebeat > k8s-app: filebeat > template: > metadata: > labels: > name: filebeat > k8s-app: filebeat > spec: > containers: > - name: filebeat-container > image: docker-registry-host/shared/filebeat > volumeMounts: > - mountPath: /var/lib/docker/containers > name: filebeat-log-source > - name: filebeat-config > mountPath: /opt/filebeat > - name: varlogcont > mountPath: /var/log/containers > readOnly: true > - name: varlogpods > mountPath: /var/log/pods > readOnly: true > imagePullSecrets: > - name: regcred > volumes: > - name: filebeat-log-source > hostPath: > path: /var/lib/docker/containers > type: DirectoryOrCreate > - name: filebeat-config > configMap: > defaultMode: 0644 > name: filebeat-config-json > - name: varlogcont > hostPath: > path: /var/log/containers > type: DirectoryOrCreate > - name: varlogpods > hostPath: > path: /var/log/pods > type: DirectoryOrCreate > --- > apiVersion: rbac.authorization.k8s.io/v1beta1 > kind: ClusterRoleBinding > metadata: > name: filebeat > subjects: > - kind: ServiceAccount > name: default > namespace: default > roleRef: > kind: ClusterRole > name: filebeat > apiGroup: rbac.authorization.k8s.io > --- > apiVersion: rbac.authorization.k8s.io/v1beta1 > kind: ClusterRole > metadata: > name: filebeat > labels: > k8s-app: filebeat > rules: > - apiGroups: [""] # "" indicates the core API group > resources: > - namespaces > - pods > verbs: > - get > - watch > - list > --- > apiVersion: v1 > kind: ServiceAccount > metadata: > name: filebeat > namespace: default > labels: > k8s-app: filebeat > > ``` The log messages are: > **log-messages** > > ... > 2019-07-25T03:23:53.420Z WARN [cfgwarn] kubernetes/kubernetes.go:51 BETA: The kubernetes autodiscover is beta > 2019-07-25T03:23:53.421Z INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-daemon-srjr4 and namespace default to discover kubernetes node > 2019-07-25T03:23:53.423Z ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: %!(EXTRA string=performing request: Get [https://10.96.0.1:443/api/v1/namespaces/default/pods/filebeat-daemon-srjr4:](https://10.96.0.1:443/api/v1/namespaces/default/pods/filebeat-daemon-srjr4:) dial tcp 10.96.0.1:443: connect: connection refused) > 2019-07-25T03:23:53.423Z INFO autodiscover/autodiscover.go:105 Starting autodiscover manager > 2019-07-25T03:23:53.423Z INFO kubernetes/watcher.go:180 kubernetes: Performing a resource sync for \*v1.PodList > 2019-07-25T03:23:53.424Z ERROR kubernetes/watcher.go:183 kubernetes: Performing a resource sync err performing request: Get [https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:](https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:) dial tcp 10.96.0.1:443: connect: connection refused for \*v1.PodList > 2019-07-25T03:23:53.424Z ERROR kubernetes/kubernetes.go:127 Error starting kubernetes autodiscover provider: performing request: Get [https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:](https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&resourceVersion=0:) dial tcp 10.96.0.1:443: connect: connection refused > 2019-07-25T03:24:23.418Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":33}},"total":{"ticks":60,"time":{"ms":63},"value":60},"user":{"ticks":30,"time":{"ms":30}}},"info":{"ephemeral\_id":"03762f04-9dd8-416e-829c-d9bcba1fde01","uptime":{"ms":30020}},"memstats":{"gc\_next":4194304,"memory\_alloc":1571712,"memory\_total":3527480,"rss":19968000}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":2},"load":{"1":5.7,"15":4.33,"5":4.47,"norm":{"1":2.85,"15":2.165,"5":2.235}}}}}} While investigating this issue, we found the temporary solution on the StackOverflow: > but the solution depends on sleep, that we do not predict on different infrastructures. The suggested solution is to retry on start in the k8s client. Also, we found these issues on Beats GitHub about current kubernetes client in > > > Hi filebeat experts, > > We still have the memory leak problem on filbeat 6.5.1. …Can anyone help us to resolved it or give us some suggestions? > > We once used filebeat 6.0 in our production environment but found that there was a memory leak problem. Then we upgraded it to version 6.5.1, in which the memory leak problem is said to be fixed according the filebeat community. Sadly our pressure test on version 6.5.1 is not satisfied and this memory leak problem seems not perfectly fixed. Below are the steps how we run our tests: > > 1).Every 10 seconds, a log file with 1.6M is generated on two nodes separately, one is deployed with filebeat 6.0, another is deployed with filebeat 6.5.1 > > We only keep the latest 200 log files on a node and clear the files generated early to free some disk space. > > After running for a whole day, both filebeat 6.5.1 and filebeat 6.0 consume some memories. And filebeat 6.5.1 consumes more that filebeat 6.0 does. > > We highly suspect that this memory leak problem has something to so with the logs that are cleared in the step > 2). Seems that some filebeat harvesters are created to gather the logs that are unluckily cleared before they are harvested. Those harvesters with no logs to harvest throw exceptions and cause the memory leak problem. > > Would anyone could help us to look into this issue? > > Thanks in advance. > filebeat6.5.1 yml: > !\[image\](https://user-images.githubusercontent.com/23073615/49261465-517c7380-f47d-11e8-9e2f-41338a66cea0.png) > !\[image\](https://user-images.githubusercontent.com/23073615/49261483-5e00cc00-f47d-11e8-89ac-5ad706b4f236.png) > !\[image\](https://user-images.githubusercontent.com/23073615/49261488-6822ca80-f47d-11e8-9f13-1ccb9c8f3f09.png) > !\[image\](https://user-images.githubusercontent.com/23073615/49261492-707b0580-f47d-11e8-969a-7ac880ee006f.png) > !\[image\](https://user-images.githubusercontent.com/23073615/49261508-84266c00-f47d-11e8-8131-5a235f52652b.png) > !\[image\](https://user-images.githubusercontent.com/23073615/49261533-9dc7b380-f47d-11e8-804b-15e014bec67a.png) and > > > This issue to collect a list of technical debt issue / tasks which should be han…dled after 7.0.0-beta1 is shipped: > > \* \[\] Clean up \`make update\` target and move it to \`mage update\` by using Go commands for all the commands > \* \[\] Remove \`metricbeat\` hacks in update command inside libbeat/scripts/Makefile > \* \[X\] Update all modules / metricsets to use v2 reporter method instead of Fetch. The goal is to make using module / global fields easier. https://github.com/elastic/beats/issues/10774 > \* \[\] Clean up code around dashboard / kibana to remove support for older 5.x versions. > \* \[\] Simplify system tests by abstracting common components (see ES module as example) > \* \[\] Simplify go integration tests by reusing common components (see ES module as example) > \* \[x\] Move docs check to golang from python. This should allow most of the system tests to not require ES anymore but only try to run the module (without service running). https://github.com/elastic/beats/pull/11127 > \* \[x\] Implement a http test helper that can be used to fetch data, map it and compare output. This will allow to test various inputs to outputs without having to run the service. Similar to the .expected files in Filebeat modules. > \* \[x\] Move to kubernetes client-go https://github.com/elastic/beats/issues/10337 > \* \[x\] Add missing dashboards: https://github.com/elastic/beats/issues/10594 > > > We are currently using https://github.com/ericchiang/k8s client, as a way to hav…e a lightweight client for Kubernetes features (autodiscover and metadata processor). We sometimes see issues with the client, being the main ones: > > \- Error parsing output from API server > \- Error while performing watch call (EOF), where we manually handle the reconnect (ie https://github.com/elastic/beats/issues/9078) > > Kubernetes \[client-go\](https://github.com/kubernetes/client-go) handles these issues for us, the only penalty is it increases the resulting binary size (+~30MB). > > I think benefits overweight the size penalty My questions are there are plans to fix those issues and on which dates? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2019, 3:43am UTC](https://discuss.elastic.co/t/filebeat-errors-while-connecting-to-kubernetes-api-on-start-filebeat-pod/192158/2 "2019-08-22T03:43:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.