DNS lookup failure "elasticsearch":

Elastic Stack Beats
1

I have elasticsearch and kibana running in local k8s cluster (using kind) and they are up and running. I deployed filebeat using image version 7.5.2 and it is up and running but I see some errors in its logs that it cannot find elasticsearch.

2020-01-22T21:27:46.380Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get http://elasticsearch:9200: lookup elasticsearch on 10.96.0.10:53: no such host
2020-01-22T21:27:46.380Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 39 reconnect attempt(s)
2020-01-22T21:27:46.381Z INFO [publisher] pipeline/retry.go:196 retryer: send unwait-signal to consumer
2020-01-22T21:27:46.381Z INFO [publisher] pipeline/retry.go:198 done
2020-01-22T21:27:46.381Z INFO [publisher] pipeline/retry.go:173 retryer: send wait signal to consumer
2020-01-22T21:27:46.381Z INFO [publisher] pipeline/retry.go:175 done
2020-01-22T21:27:46.386Z WARN transport/tcp.go:53 DNS lookup failure "elasticsearch": lookup elasticsearch on 10.96.0.10:53: no such host
2020-01-22T21:28:00.358Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2950},"total":{"ticks":4680,"time":{"ms":40},"value":4680},"user":{"ticks":1730,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"a3b4c7b1-6efb-4f26-abd9-8ac6ca707185","uptime":{"ms":1680052}},"memstats":{"gc_next":54458480,"memory_alloc":28274464,"memory_total":85599704},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":6,"running":5}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":4117,"retry":50}}},"registrar":{"states":{"current":3}},"system":{"load":{"1":0.32,"15":0.36,"5":0.34,"norm":{"1":0.08,"15":0.09,"5":0.085}}}}}}

Any ideas? How can I change domain name to localhost?

I figured that out how to change it but there is another error:

ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp [::1]:9200: connect: cannot assign requested address

Here is the output of service. I also tried this one (quickstart-es-http) that should be the correct one but still got the same error:

kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 443/TCP 126m
default quickstart-es-default ClusterIP None 43m
default quickstart-es-http ClusterIP 10.109.76.122 9200/TCP 43m
default quickstart-kb-http ClusterIP 10.110.51.207 5601/TCP 40m
elastic-system elastic-webhook-server ClusterIP 10.96.97.156 443/TCP 45m
kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 126m

2

Hi @Adib_Rastegarnia :slight_smile:

Unfortunately we haven't tried kind but maybe you can also upload your config and maybe someone can find something.

From what I can see, it looks like some simple error. Not necessarly related with the DNS but maybe an http over https in some field in the config. Some localhost in a place where it should be an IP or a different network name.

3

Hi @Mario_Castro

Thanks for your response. I am following quick start guide and didn't change something significant. I think the original problem is solved by changing the host from http to https but another problem comes up and it is connection refused problem.

2020-01-23T14:00:36.042Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://quickstart-es-http:9200)): Get https://quickstart-es-http:9200: dial tcp 23.217.138.110:9200: connect: connection refused
2020-01-23T14:00:36.042Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://quickstart-es-http:9200)) with 3 reconnect attempt(s)
2020-01-23T14:00:36.042Z INFO [publisher] pipeline/retry.go:196 retryer: send unwait-signal to consumer
2020-01-23T14:00:36.042Z INFO [publisher] pipeline/retry.go:198 done
2020-01-23T14:00:36.042Z INFO [publisher] pipeline/retry.go:173 retryer: send wait signal to consumer
2020-01-23T14:00:36.042Z INFO [publisher] pipeline/retry.go:175 done

The solutions out there are referring to modifying network.host field in elasticsearch yaml file. what do you suggest?

4

@Mario_Castro

Actually, I am getting the DNS error again and have no idea. I also tested on Google cloud and got the same error. What I changed previously was this line by adding https:

I tried with IP address of service using https but got the certificate issue, is it possible to bypass it?

ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://10.109.76.122:9200)): Get https://10.109.76.122:9200: x509: cannot validate certificate for 10.109.76.122 because it doesn't contain any IP SAN
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.