# Filebeat 7.4.0 does not recover when it fails to connect with k8s API **URL:** \ **Category:** Beats\ **Tags:** docker\ **Created:** [October 19, 2019, 11:14pm UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327 "2019-10-19T23:14:54Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![GreenKnight15](https://avatars.discourse-cdn.com/v4/letter/g/c67d28/32.png) [@GreenKnight15](https://discuss.elastic.co/u/GreenKnight15)\ **Post date:** [October 19, 2019, 11:14pm UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327/1 "2019-10-19T23:14:54Z")
I am using the filebeat elastic helm chart [https://github.com/elastic/helm-charts/tree/master/filebeat](https://github.com/elastic/helm-charts/tree/master/filebeat) under an Istio service mesh. As of filebeat 7.4.0 with the new k8s client [Update kubernetes watcher to use official client-go libraries by vjsamuel · Pull Request #13051 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/13051) filebeat starts faster than the Istio side car which blocks outbound requests to the k8s API. As a result filebeat never recovers the k8s connection and I lose all k8s meta data on my log packets. > │ 2019-10-17T20:33:29.733Z ERROR kubernetes/util.go:85 kubernetes: Querying for pod failed with error: Get [https://10.100.0.1:443/api/v1/namespaces/bootstrap/pods/bootstrap-filebeat-x-pj8n9:](https://10.100.0.1:443/api/v1/namespaces/bootstrap/pods/bootstrap-filebeat-x-pj8n9:) dial tcp 10.100.0.1:443: connect: connection refused │ > │ E1017 20:33:29.734464 1 reflector.go:125] [github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235:](http://github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235:) Failed to list \*v1.Pod: Get [https://10.100.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&limit=500&resour](https://10.100.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&limit=500&resour) │ > │ ceVersion=0: dial tcp 10.100.0.1:443: connect: connection refused I found a workaround by applying a sleep before starting filebeat, but I don't want it to be permanent. A similar issue is describe in [Kubernetes autodiscover provider fails silently if it can't connect to k8s · Issue #13081 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/13081) Will there be a retry or exponential back off added in upcoming versions? ---
**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\ **Post date:** [October 21, 2019, 10:44am UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327/2 "2019-10-21T10:44:48Z")
Hi @GreenKnight15, let me try to reproduce / follow the code. I'm not sure of if beats have a policy of retrying or failing, looks like it is mostly mandated by the library being used. If you feel so, open an issue where that behaviour can be discussed while I guess out how it currently works. ---
**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\ **Post date:** [October 21, 2019, 11:25am UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327/3 "2019-10-21T11:25:05Z")
Hi again @GreenKnight15 , there are 2 potential _features_ affected by this issue - autodiscover - add kubernetes metadata - a watcher that is setup for kubernetes metricsets to _enrich_ events I think we can do something there, but we need the to get the product designers involved, can you please create a GH issue? thanks! ---
**Author:** ![GreenKnight15](https://avatars.discourse-cdn.com/v4/letter/g/c67d28/32.png) [@GreenKnight15](https://discuss.elastic.co/u/GreenKnight15)\ **Post date:** [October 21, 2019, 3:31pm UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327/4 "2019-10-21T15:31:02Z")
I wend ahead and opened a GH ticket > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 18, 2019, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-7-4-0-does-not-recover-when-it-fails-to-connect-with-k8s-api/204327/5 "2019-11-18T17:31:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.