Crontab, ikuturso-indexer logstash 6.2 and open source threat intelligence


#1

I have an ikuturso/logstash-indexer:6.2 container which receives the data from rabbit-mq. Within the container I have a scheduled crontab script which regularly downloads the open source feed Minemeld (Paloalto Threat Intelligence) and accordingly updates the logstash configuration files (several .conf).

The container starts with the following command:
logstash -f /usr/share/logstash/pipeline -r --config.reload.interval 60s --log.level warn

This way you can dynamically reload the logstash configuration files.

_Issue: logstash crashes with the following logs:_

17/10/2018 14:40:10[2018-10-17T12:40:10,008][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)
...

17/10/2018 14:41:45[2018-10-17T12:41:45,809][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error}
...

17/10/2018 14:47:18[2018-10-17T12:47:18,955][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Connection reset)

17/10/2018 14:47:39[2018-10-17T12:47:39,022][WARN ][com.rabbitmq.client.impl.ForgivingExceptionHandler] An unexpected connection driver error occured (Exception message: Socket closed)

17/10/2018 14:53:18[2018-10-17T12:53:18,135][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqp://admin:XXXXXX@localhost:5672/", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)}

17/10/2018 14:53:23[2018-10-17T12:53:23,090][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:28[2018-10-17T12:53:28,085][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:28[2018-10-17T12:53:28,090][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

17/10/2018 14:53:33[2018-10-17T12:53:33,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 14:53:38[2018-10-17T12:53:38,086][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{}}

17/10/2018 15:00:00crond[15513]: USER root pid 15872 cmd /usr/share/logstash/config/updatefeed.sh > /dev/null 2>&1

17/10/2018 15:03:32java.lang.OutOfMemoryError: Java heap space

17/10/2018 15:03:32Dumping heap to java_pid1.hprof ...

17/10/2018 15:03:32Unable to create java_pid1.hprof: Permission denied

17/10/2018 15:27:29[2018-10-17T13:27:29,198][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Reload/pipeline_id:main, :exception=>"Java::JavaLang::OutOfMemoryError", :message=>"Java heap space", :backtrace=>[]}

17/10/2018 15:27:29[2018-10-17T13:27:29,668][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::OutOfMemoryError for LogStash::PipelineAction::Reload/pipeline_id:main", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:27:in create'", "/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:67:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

17/10/2018 15:27:54[2018-10-17T13:27:54,004][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space


(jack) #2

What do you want to achive? Do you want to reload the configuration files without stopping and restarting the container?


#3

Yes, the goal is to modify the configuration files via crontab script without stopping and restarting logstash-indexer container...

After the crash logstash restarts... but only after 40 circa minutes and crond service within the container is no more active.... so basically, cause these errors, I can not integrate the threat intelligence and automate the process.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.