I'm new to ELK and beats, I set up filebeat and merticbeat on 2 linux servers to send logs/metrics to Logstash on another server in our AWS Clould.
On one of the beat servers, beats work fine and send data to logstash and shows on Kibana. But one another one, the beats look fine, but no data showing on Kibana.
I checked the beats logs (both filebeat and metricbeat logs), I kept getting " tcp xx.xx.xx.xx:5044: i/o timeout", xx.xx.xx.xx is the IP of the ELK server. I thought it was network issue, but could not find anything wrong on network, opened a ticket to AWS support, they looked at this and could not find network issue either. They said this might be the beats configuration related.
I used the same configuration for both Linux servers, but one works, the other one doesn't.
I'm not sure how to attached the filebeat.yml and log, the only errors is:
ERROR pipeline/output.go:74 Failed to connect: dial tcp xx.xx.xx.xx:5044: i/o timeout
On the filebeat.yml, I used almost all the detaults except commented out the Elasticsearch out put and uncommented the Logstash output to point to the ELK server IP...
telnet xx.xx.xx.xx 5044 doesn't work on this server.. but works on the other beats server.
This may be network related but just wanted to make sure the configuration on the beats are fine.
All beats and ELK we used are version 6.3.2
Please help and let me know what it could be...