Filebeat can not talk to ELK on AWS EKS

Elastic Stack Elasticsearch
1

Hello Everyone,

I am using the follow doc to setup the filebeat on EKS.

curl -L -O https://raw.githubusercontent.com/elastic/beats/8.7/deploy/kubernetes/filebeat-kubernetes.yaml

Default Setting does not work

  • name: ELASTICSEARCH_HOST
    value: elasticsearch
  • name: ELASTICSEARCH_PORT
    value: "9200"

for ELASTICSEARCH_HOST.

ELK on AWS EKS is setup from the following doc.

Here is the error when configure the host to one of the elastic service quickstart-es-http
after elasticsearch setting does not work for host.

By configuring the ingress service for Elastic, it appears that elastic 9200 is using tls. But Filebeat 8.7 is trying to connect to it with http, not sure whether it is the problem.

Thank you very much for your help in advance!

{"log.level":"info","@timestamp":"2023-04-07T20:19:29.068Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(http://quickstart-es-http:9200)) with 4 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2023-04-07T20:19:29.072Z","log.logger":"transport","log.origin":{"file.name":"transport/tcp.go","file.line":52},"message":"DNS lookup failure \"quickstart-es-http\": lookup quickstart-es-http on 172.20.0.10:53: no such host","service.name":"filebeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-04-07T20:19:29.072Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":38},"message":"Error dialing lookup quickstart-es-http on 172.20.0.10:53: no such host","service.name":"filebeat","network":"tcp","address":"quickstart-es-http:9200","ecs.version":"1.6.0"}

Thank you very much!

2

Just want to add. I am able to send the filebeat log to Elastic's ingress URL on ALB. But I would like to try to send the logs internally. This is where I have issues. Thank you very much!

3

The error shows that quickstart-es-http is not resolved by filebeat. Maybe you should use service FQDN like quickstart-es-http.namespace.svc.cluster.local.

Regarding connecting with TLS, this document can help you - Configure SSL | Filebeat Reference [8.7] | Elastic

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.