I have installed filebeat on 172-31-0-204 and logstash on 172-31-0-207. I created the ES domain in AWS and gave that domain in logstash config file on 172-31-0-207. The issue is when I give below command in logstash server to push logs from logstash to kibanna
"/opt/logstash/bin/logstash -f logstash.conf", the log processing is stopping at below text:
source":"/var/log/messages","host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}
That means the filebeat server stops pushing logs to logstash server after few mins.
So here I need to restart filebeat on 172-31-0-204 and then again need to run "/opt/logstash/bin/logstash -f logstash.conf
"cmd on logstash host so that the updated logs can push to kibana. And after few seconds the logs processing stops again at below text
{"message":"May 4 11:21:07 ip-172-31-0-204 dhclient[2041]: bound to 172.31.0.204 -- renewal in 1542 seconds.","@version":"1","@timestamp":"2016-05-04T11:24:52.428Z","count":1,"fields":null,"beat":{"hostname":"ip-172-31-0-204","name":"ip-172-31-0-204"},"type":"log","input_type":"log","offset":117296,"source":"/var/log/messages","host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}
what errors are there in the beats.log and what errors are in the logstash.log? Need to provide more information for us to help. Also the config for logstash and filebeat would help.
Actually the logs are only pushed from filebeat server to logstash server when I restart filebeat and after sometime log processing stops at below lines @timestamp":"2016-05-04T12:46:14.670Z","source":"/var/log/messages","count":1,"beat":{"hostname":"ip-172-31-0-204","name":"ip-172-31-0-204"},"offset":120152,"type":"log","input_type":"log","fields":null,"host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}
Can you turn the verbosity up on the logging for both filebeat and logstash? Still not much to go on to figure out what is causing things to "stop" Any error message in the filebeat logs?
EADDRINUSE generally occurs if the port is already in use. A mis-configuration in logstash where logstash starts listening on the port 2x or is run multiple time can also cause this.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.