Need help on Logstash server

Elastic Stack Logstash
#1

I have installed filebeat on 172-31-0-204 and logstash on 172-31-0-207. I created the ES domain in AWS and gave that domain in logstash config file on 172-31-0-207. The issue is when I give below command in logstash server to push logs from logstash to kibanna
"/opt/logstash/bin/logstash -f logstash.conf", the log processing is stopping at below text:
source":"/var/log/messages","host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}

That means the filebeat server stops pushing logs to logstash server after few mins.

So here I need to restart filebeat on 172-31-0-204 and then again need to run "/opt/logstash/bin/logstash -f logstash.conf
"cmd on logstash host so that the updated logs can push to kibana. And after few seconds the logs processing stops again at below text
{"message":"May 4 11:21:07 ip-172-31-0-204 dhclient[2041]: bound to 172.31.0.204 -- renewal in 1542 seconds.","@version":"1","@timestamp":"2016-05-04T11:24:52.428Z","count":1,"fields":null,"beat":{"hostname":"ip-172-31-0-204","name":"ip-172-31-0-204"},"type":"log","input_type":"log","offset":117296,"source":"/var/log/messages","host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}

please help

#2

what errors are there in the beats.log and what errors are in the logstash.log? Need to provide more information for us to help. Also the config for logstash and filebeat would help.

#3

Error in the /var/log/logstash.log

, "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90:inrun'", "org/jruby/RubyProc.java:281:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95:inrun'", "org/jruby/RubyProc.java:281:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:ininitialize'"], :level=>:warn}

Actually the logs are only pushed from filebeat server to logstash server when I restart filebeat and after sometime log processing stops at below lines
@timestamp":"2016-05-04T12:46:14.670Z","source":"/var/log/messages","count":1,"beat":{"hostname":"ip-172-31-0-204","name":"ip-172-31-0-204"},"offset":120152,"type":"log","input_type":"log","fields":null,"host":"ip-172-31-0-204","tags":["beats_input_codec_plain_applied"]}

#4

logstash.conf:

input {

beats {

  port => 5000

}

}

output {
stdout {
codec => json
}

   elasticsearch {
          manage_template => false
          hosts => 'http://search-escluster-npqf7ydm...:80'

}

}

Filebeat.conf:

filebeat:

prospectors:

-

  paths:

    - /var/log/httpd/access.log

    - /var/log/messages

  input_type: log

output:

logstash:

hosts: ["172.31.0.207:5000"]

console:

pretty: true

shipper:

logging:

files:

rotateeverybytes: 10485760 # = 10MB

To get logs pushed to kibana every time I need to restart logstash and filebeat. Please suggest me solution for this

#5

Could someone please provide me an update on this

#6

Can you turn the verbosity up on the logging for both filebeat and logstash? Still not much to go on to figure out what is causing things to "stop" Any error message in the filebeat logs?

#7

Error in the /var/log/logstash.log

{:timestamp=>"2016-05-04T12:30:23.509000+0000", :message=>"An unexpected error occurred!", :error=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :class=>"Errno::EADDRINUSE", :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in initialize'", "org/jruby/RubyIO.java:853:innew'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.7/lib/lumberjack/beats/server.rb:51:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.7/lib/logstash/inputs/beats.rb:119:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:322:in start_inputs'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:321:in start_inputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172:instart_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=>:warn}

#8

could you please provide an update on this

#9

is there something else listening on port 5000?

EADDRINUSE generally occurs if the port is already in use. A mis-configuration in logstash where logstash starts listening on the port 2x or is run multiple time can also cause this.

#10