Hello. I´m new with ELK. I have an ELK cluster with the following:
Node1: elasticsearch, logstash, kibana || 32GB RAM || -Xms12g, -Xmx12g
Node2: elasticsearch, logstash || 32GB RAM || -Xms12g, -Xmx12g
Node3,4,5: masterdata
When monitoring with Zabbix Node1 Logstash ports seems to go down after a wile (ports 5043 to 5047). Node2 its ok.
I see the following on Node1 /var/log/logstash/logstash-plain.log
[2019-07-24T11:18:05,622][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.InterruptedException,
:backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(o
rg/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/l
ogstash/java_pipeline.rb:239)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.
call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2019-07-24T11:18:05,773][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-07-24T11:18:19,855][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-24T00:25:58,582][INFO ][org.logstash.beats.Server] Starting server on port: 5046
[2019-07-24T00:25:58,583][INFO ][org.logstash.beats.Server] Starting server on port: 5045
[2019-07-24T00:25:58,585][INFO ][org.logstash.beats.Server] Starting server on port: 5047
[2019-07-24T00:25:58,586][INFO ][org.logstash.beats.Server] Starting server on port: 5043
[2019-07-24T00:25:58,586][INFO ][org.logstash.beats.Server] Starting server on port: 5044
After 5 minutes the ports are up again, but on Node2 this is not happening.
Can you help me with this?.
Thanks!!