I am using the filebeat elastic helm chart https://github.com/elastic/helm-charts/tree/master/filebeat under an Istio service mesh.
As of filebeat 7.4.0 with the new k8s client https://github.com/elastic/beats/pull/13051 filebeat starts faster than the Istio side car which blocks outbound requests to the k8s API. As a result filebeat never recovers the k8s connection and I lose all k8s meta data on my log packets.
│ 2019-10-17T20:33:29.733Z ERROR kubernetes/util.go:85 kubernetes: Querying for pod failed with error: Get https://10.100.0.1:443/api/v1/namespaces/bootstrap/pods/bootstrap-filebeat-x-pj8n9: dial tcp 10.100.0.1:443: connect: connection refused │
│ E1017 20:33:29.734464 1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1.Pod: Get https://10.100.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dlocalhost&limit=500&resour │
│ ceVersion=0: dial tcp 10.100.0.1:443: connect: connection refused
I found a workaround by applying a sleep before starting filebeat, but I don't want it to be permanent.
A similar issue is describe in https://github.com/elastic/beats/issues/13081
Will there be a retry or exponential back off added in upcoming versions?