I'm running into a problem with java.util.concurrent.ScheduledThreadPoolExecutor.
Setup details:
logstash 8.5.1
Logstash was installed via RPM
Linux server1 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Sat Aug 27 02:44:20 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
I have 5 different pipelines configured on logstash with one using inputs for eventhub, and 4 using s3.
Until recently everything was working correct, however I've run into the following problem:
[2023-02-20T19:07:28,352][ERROR][logstash.javapipeline ][pipeline-<REDACTED>][52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:pipeline-<REDACTED>
Plugin: <LogStash::Inputs::S3 access_key_id=>"<REDACTED>", bucket=>"<REDACTED>", secret_access_key=><password>, id=>"52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e", region=>"<REDACTED>", additional_settings=>{"force_path_style"=>"true"}, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e41b9c6d-332b-4509-be39-cdfe8fb24dfe", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
Error: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@13021824[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40ad76f5[Wrapped task = org.jruby.ext.timeout.Timeout$TimeoutTask@4b378a4c]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6748cc5b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
Exception: Java::JavaUtilConcurrent::RejectedExecutionException
Stack: java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
org.jruby.ext.timeout.Timeout.yieldWithTimeout(Timeout.java:145)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:118)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:92)
org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(Timeout$INVOKER$s$timeout.gen)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$connect$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:949)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$do_start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:934)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:929)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0$__VARARGS__(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:919)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyBasicObject.send(RubyBasicObject.java:1733)
org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
org.jruby.ir.targets.indy.InvokeSite.fail(InvokeSite.java:237)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.delegate.RUBY$method$method_missing$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/delegate.rb:83)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
<SNIP>
When this occurs all pipelines fail, is there a recommended way to troubleshoot java.util.concurrent.ScheduledThreadPoolExecutor issues? I can reproduce this issue, it seems to happen every 15-30 minutes
Please let me know if there's something I can do to troubleshoot this, I don't see anything that sticks out even when I turn debug on.
I'm going through right now and limiting the amount of pipelines I have to see if that has any affect on the issue.