Crontab, ikuturso-indexer logstash 6.2 and open source threat intelligence

I have an ikuturso/logstash-indexer:6.2 container which receives the data from rabbit-mq. Within the container I have a scheduled crontab script which regularly downloads the open source feed Minemeld (Paloalto Threat Intelligence) and accordingly updates the logstash configuration files (several .conf).

The container starts with the following command:
logstash -f /usr/share/logstash/pipeline -r --config.reload.interval 60s --log.level warn

This way you can dynamically reload the logstash configuration files.

_Issue: logstash crashes with the following logs:_

17/10/2018 14:40:10[2018-10-17T12:40:10,008][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)
...

17/10/2018 14:41:45[2018-10-17T12:41:45,809][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error}
...

17/10/2018 14:47:18[2018-10-17T12:47:18,955][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)

17/10/2018 14:47:39[2018-10-17T12:47:39,022][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Socket closed)

17/10/2018 14:53:18[2018-10-17T12:53:18,135][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)}

17/10/2018 14:53:23[2018-10-17T12:53:23,090][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:28[2018-10-17T12:53:28,085][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:28[2018-10-17T12:53:28,090][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

17/10/2018 14:53:33[2018-10-17T12:53:33,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:38[2018-10-17T12:53:38,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 15:00:00crond[15513]: USER root pid 15872 cmd /usr/share/logstash/config/updatefeed.sh > /dev/null 2>&1

17/10/2018 15:03:32java.lang.OutOfMemoryError: Java heap space

17/10/2018 15:03:32Dumping heap to java_pid1.hprof ...

17/10/2018 15:03:32Unable to create java_pid1.hprof: Permission denied

17/10/2018 15:27:29[2018-10-17T13:27:29,198][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Reload/pipeline_id:main, :exception=>"Java::JavaLang::OutOfMemoryError", :message=>"Java heap space", :backtrace=>[]}

17/10/2018 15:27:29[2018-10-17T13:27:29,668][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::OutOfMemoryError for LogStash::PipelineAction::Reload/pipeline_id:main", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:27:in create'", "/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:67:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

17/10/2018 15:27:54[2018-10-17T13:27:54,004][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space

What do you want to achive? Do you want to reload the configuration files without stopping and restarting the container?

Yes, the goal is to modify the configuration files via crontab script without stopping and restarting logstash-indexer container...

After the crash logstash restarts... but only after 40 circa minutes and crond service within the container is no more active.... so basically, cause these errors, I can not integrate the threat intelligence and automate the process.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.