Metricbeat in Kubernetes fails to find Elasticsearch host

[bladerunner512]

metricbeat 6.2.3/elasticsearch 6.2.2/kubernetes 1.9
error:
ERROR pipeline/output.go:74 Failed to connect: Get http://elasticsearch:9200: lookup elasticsearch on 148.93.51.11:53: no such host

config values:
ELASTICSEARCH_HOST=elasticsearch
ELASTICSEARCH_PORT=9200

Where does '148.93.51.11:53' come from? No such IP in system?
Works fine if host is changed to Kubernetes internal Cluster IP created for Elasticsearch service.

Interestingly, Filebeat has a similar config, but is able to resolve elasticsearch:9200 host

[exekias (Carlos Pérez Aradros)]

Hi @bladedrunner512,

That IP is the DNS server, the error comes from its answer.

In order to debug why Metricbeat cannot resolve elasticsearch we would need some more info about your deploy, like Metricbeat manifests and how elasticsearch is deployed. I understand there is an elasticsearch service in the same namespace?

Best regards

[exekias (Carlos Pérez Aradros)]

On a second though, I'm guessing this is related to hostNetwork: true. Have a look to https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pods-dns-policy. You can probably add dnsPolicy: ClusterFirstWithHostNet to fix this issue.

Please let me know if that fixes the issue, we may want to put that in the official manifest.

Best regards