I have an ikuturso/logstash-indexer:6.2 container which receives the data from rabbit-mq. Within the container I have a scheduled crontab script which regularly downloads the open source feed Minemeld (Paloalto Threat Intelligence) and accordingly updates the logstash configuration files (several .conf).
The container starts with the following command:
logstash -f /usr/share/logstash/pipeline -r --config.reload.interval 60s --log.level warn
This way you can dynamically reload the logstash configuration files.
_Issue: logstash crashes with the following logs:_
17/10/2018 14:40:10[2018-10-17T12:40:10,008][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)
...
17/10/2018 14:41:45[2018-10-17T12:41:45,809][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error}
...
17/10/2018 14:47:18[2018-10-17T12:47:18,955][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)
17/10/2018 14:47:39[2018-10-17T12:47:39,022][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Socket closed)
17/10/2018 14:53:18[2018-10-17T12:53:18,135][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)}
17/10/2018 14:53:23[2018-10-17T12:53:23,090][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}
17/10/2018 14:53:28[2018-10-17T12:53:28,085][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}
17/10/2018 14:53:28[2018-10-17T12:53:28,090][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
17/10/2018 14:53:33[2018-10-17T12:53:33,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}
17/10/2018 14:53:38[2018-10-17T12:53:38,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}
17/10/2018 15:00:00crond[15513]: USER root pid 15872 cmd /usr/share/logstash/config/updatefeed.sh > /dev/null 2>&1
17/10/2018 15:03:32java.lang.OutOfMemoryError: Java heap space
17/10/2018 15:03:32Dumping heap to java_pid1.hprof ...
17/10/2018 15:03:32Unable to create java_pid1.hprof: Permission denied
17/10/2018 15:27:29[2018-10-17T13:27:29,198][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Reload/pipeline_id:main, :exception=>"Java::JavaLang::OutOfMemoryError", :message=>"Java heap space", :backtrace=>[]}
17/10/2018 15:27:29[2018-10-17T13:27:29,668][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::OutOfMemoryError
for LogStash::PipelineAction::Reload/pipeline_id:main
", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:27:in create'", "/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:67:in
add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in
with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:in
each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in
block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in
converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in
interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in
block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
17/10/2018 15:27:54[2018-10-17T13:27:54,004][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space