Logstash - A plugin had an unrecoverable error. Will restart this plugin. java.util.concurrent.ScheduledThreadPoolExecutor

I'm running into a problem with java.util.concurrent.ScheduledThreadPoolExecutor.

Setup details:
logstash 8.5.1
Logstash was installed via RPM
Linux server1 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Sat Aug 27 02:44:20 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

I have 5 different pipelines configured on logstash with one using inputs for eventhub, and 4 using s3.

Until recently everything was working correct, however I've run into the following problem:


[2023-02-20T19:07:28,352][ERROR][logstash.javapipeline    ][pipeline-<REDACTED>][52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:pipeline-<REDACTED>
  Plugin: <LogStash::Inputs::S3 access_key_id=>"<REDACTED>", bucket=>"<REDACTED>", secret_access_key=><password>, id=>"52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e", region=>"<REDACTED>", additional_settings=>{"force_path_style"=>"true"}, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e41b9c6d-332b-4509-be39-cdfe8fb24dfe", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
  Error: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@13021824[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40ad76f5[Wrapped task = org.jruby.ext.timeout.Timeout$TimeoutTask@4b378a4c]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6748cc5b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
  Exception: Java::JavaUtilConcurrent::RejectedExecutionException
  Stack: java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
org.jruby.ext.timeout.Timeout.yieldWithTimeout(Timeout.java:145)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:118)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:92)
org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(Timeout$INVOKER$s$timeout.gen)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$connect$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:949)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$do_start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:934)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:929)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0$__VARARGS__(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:919)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyBasicObject.send(RubyBasicObject.java:1733)
org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
org.jruby.ir.targets.indy.InvokeSite.fail(InvokeSite.java:237)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.delegate.RUBY$method$method_missing$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/delegate.rb:83)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
<SNIP>

When this occurs all pipelines fail, is there a recommended way to troubleshoot java.util.concurrent.ScheduledThreadPoolExecutor issues? I can reproduce this issue, it seems to happen every 15-30 minutes

Please let me know if there's something I can do to troubleshoot this, I don't see anything that sticks out even when I turn debug on.

I'm going through right now and limiting the amount of pipelines I have to see if that has any affect on the issue.

Adding the whole error:

[2023-02-20T18:42:32,054][ERROR][logstash.javapipeline    ][pipeline-<REDACTED>][52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:pipeline-<REDACTED>
  Plugin: <LogStash::Inputs::S3 access_key_id=>"<REDACTED>", bucket=>"<REDACTED>", secret_access_key=><password>, id=>"52395b7ee18dbfc7e40de048201c506443e2e2efcc2c5ec7d85ff0f1bae9a73e", region=>"<REDACTED>", additional_settings=>{"force_path_style"=>"true"}, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_0e0a9b26-43d2-4ba1-9c96-ff8ba997c03d", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", delete=>false, interval=>60, watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">
  Error: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2e3cfd28[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@37667b42[Wrapped task = org.jruby.ext.timeout.Timeout$TimeoutTask@734640e6]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@17fd54bf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
  Exception: Java::JavaUtilConcurrent::RejectedExecutionException
  Stack: java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
org.jruby.ext.timeout.Timeout.yieldWithTimeout(Timeout.java:145)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:118)
org.jruby.ext.timeout.Timeout.timeout(Timeout.java:92)
org.jruby.ext.timeout.Timeout$INVOKER$s$timeout.call(Timeout$INVOKER$s$timeout.gen)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$connect$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:949)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$do_start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:934)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:929)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.net.http.RUBY$method$start$0$__VARARGS__(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:919)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyBasicObject.send(RubyBasicObject.java:1733)
org.jruby.RubyBasicObject$INVOKER$i$send.call(RubyBasicObject$INVOKER$i$send.gen)
org.jruby.ir.targets.indy.InvokeSite.fail(InvokeSite.java:237)
usr.share.logstash.vendor.jruby.lib.ruby.stdlib.delegate.RUBY$method$method_missing$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/delegate.rb:83)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.connection_pool.RUBY$method$start_session$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/connection_pool.rb:307)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.connection_pool.RUBY$method$session_for$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/connection_pool.rb:100)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$session$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:128)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$transmit$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:76)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.net_http.handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/net_http/handler.rb:50)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.content_length.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/content_length.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.request_callback.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/request_callback.rb:85)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:132)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:63)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_host_id.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.xml.error_handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/xml/error_handler.rb:10)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.transfer_encoding.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.helpful_socket_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.s3_signer.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:110)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.redirects.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/redirects.rb:20)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.retry_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/retry_errors.rb:360)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.http_checksum.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/http_checksum.rb:19)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.endpoint_pattern.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.accelerate.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:67)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.checksum_algorithm.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.dualstack.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:41)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.expect_100_continue.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.bucket_name_restrictions.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.arn.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:62)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.rest.handler.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/rest/handler.rb:10)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.recursion_detection.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.user_agent.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/user_agent.rb:13)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.endpoint_discovery.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.endpoint.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/endpoint.rb:47)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.param_validator.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/param_validator.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.arn.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:88)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.raise_response_errors.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/raise_response_errors.rb:16)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.sse_cpk.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.dualstack.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:27)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.plugins.accelerate.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:56)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.checksum_algorithm.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.jsonvalue_converter.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.idempotency_token.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.param_converter.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/param_converter.rb:26)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.request_callback.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/request_callback.rb:71)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.plugins.response_paging.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/plugins/response_paging.rb:12)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.plugins.response_target.RUBY$method$call$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/plugins/response_target.rb:24)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.seahorse.client.request.RUBY$method$send_request$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/seahorse/client/request.rb:72)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.client.RUBY$method$list_objects_v2$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/client.rb:8556)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_s3_minus_1_dot_114_dot_0.lib.aws_minus_sdk_minus_s3.bucket.RUBY$block$objects$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/bucket.rb:912)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$block$non_empty_batches$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:101)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:52)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.RubyGenerator.each(RubyGenerator.java:99)
org.jruby.RubyGenerator$INVOKER$i$0$0$each.call(RubyGenerator$INVOKER$i$0$0$each.gen)
org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)
org.jruby.RubyClass.finvoke(RubyClass.java:502)
org.jruby.runtime.Helpers.invoke(Helpers.java:645)
org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:387)
org.jruby.RubyEnumerator.__each__(RubyEnumerator.java:396)
org.jruby.RubyEnumerator.each(RubyEnumerator.java:392)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.aws_minus_sdk_minus_core_minus_3_dot_151_dot_0.lib.aws_minus_sdk_minus_core.resources.collection.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/aws-sdk-core-3.151.0/lib/aws-sdk-core/resources/collection.rb:58)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$list_new_files$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:141)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$process_files$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:183)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:130)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.Block.call(Block.java:147)
org.jruby.RubyProc.call(RubyProc.java:272)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.stud_minus_0_dot_0_dot_23.lib.stud.interval.RUBY$method$interval$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/interval.rb:20)
usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_aws_minus_7_dot_1_dot_0.lib.logstash.inputs.s3.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.1.0/lib/logstash/inputs/s3.rb:129)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:411)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:406)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)
usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:402)
org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
org.jruby.runtime.Block.call(Block.java:143)
org.jruby.RubyProc.call(RubyProc.java:309)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)
java.base/java.lang.Thread.run(Thread.java:833)

When I limited worker processes it seemed to help. The problem I have though is why would this message pop up every second for all my s3 pipelines, and put logstash into a bad state where it requires a service restart to start functioning again?

When it goes into this state I cannot even call the API to query stats. This feels like a bug.

Anyone have any suggestions? Is this a possible config issue, sizing issue, a defect?

Does your s3 bucket have a lot of files?

Yes some of the buckets I am pulling from have thousands of files. But when this happens it's not only the one pipeline that fails it's every pipeline.

In my experience, the s3 plugin has a very bad performance when working with buckets with a lot of files.

But I'm not sure if one pipeline having issues would impact the others using the same input plugin. Are you using multiple pipelines with the pipelines.yml file? Can you share your pipelines?

I had a lot of issues trying to use the s3 input to Cloudtrail logs from a bucket and in the end I gave up and wrote a custom script to download the files. I don't know if you are trying to get custom logs or logs created by some AWS service, but in the end the logic is the same.

There are a couple of issues opened on github about this, but they are all stalled.

Are you able to use some prefix on your buckets to reduce the number of objects to list? Maybe using a prefix would help.

I'm actually doing prefixing on some of my buckets right now. But I cannot apply more prefixing otherwise I will start to lose data I want to collect.

Every s3 has a different pipeline configured for it. What I've found is the issue will occur on one of the pipes, every s3 pipeline will throw the same error and the non-s3 fail with no error. The service enters a bad state. The only way to recover is to restart the service. The built in logstash api doesn't even respond when it fails like this.

As a work around I started using:

 pipeline.workers: 

it seemed to resolve it from occurring so frequently but it keeps happening.

Here's my pipeline:

- pipeline.id: pipeline-aws-auth
  path.config: "/etc/logstash/conf.d/pipeline-aws-auth/*.conf"
  pipeline.workers: 2

- pipeline.id: pipeline-aws-s3access
  path.config: "/etc/logstash/conf.d/pipeline-s3access/*.conf"
  pipeline.workers: 1

- pipeline.id: pipeline-eventhub
  path.config: "/etc/logstash/conf.d/pipeline-eventhub/*.conf"
  pipeline.workers: 2

- pipeline.id: pipeline-aws-prod
  path.config: "/etc/logstash/conf.d/pipeline-aws-prod/*.conf"
  pipeline.workers: 1

- pipeline.id: pipeline-aws-guardduty
  path.config: "/etc/logstash/conf.d/pipeline-aws-guardduty/*.conf"
  pipeline.workers: 1

I have seen this issue, you referenced. However, if you wait long enough for it to enumerate the files it seems to function fine. Actually the only reason I know this is because I switch debug on to make sure it wasn't hung.

Not sure what you could do now, it looks like there is some issue in the plugin that impacts the entire logstash process.

But there are already some know issues in this plugin from years ago and no changes.

Can you enable SQS for your buckets? If you can, maybe you can try to move the input logic from Logstash to Filebeat as the filebeat s3 input seems to be more up to date, but it needs SQS to work.

This is unfortunate. I created a git bug report as well java.util.concurrent.ScheduledThreadPoolExecutor related to S3 input resulting in all pipeline logstash failure. · Issue #247 · logstash-plugins/logstash-input-s3 · GitHub

I was not planning on using SQS for accomplishing this. I will have to go back and review.

Any one from the Logstash team able to comment or have a look?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.