Logstash Vulnerability scanner, High CPU utilization

amaljoy · July 12, 2024, 2:28pm

There is a vulnerability scan and penetration testing done from tenable.io / nessus server in our logstash server.

This is my input plugin config in logstash.

input {
 syslog {
  id => "idsyslog"
  host => "0.0.0.0"
  port => 10514
  type => "syslog"
  codec => plain {
                    charset => "ISO-8859-1"
            }
 }
}

While the vulnerability scan happens on port 10514 its being read as logs by our logstash. Below are the messages we receive as captured by logstash.

Now this is totally fine, but the only issue we are having is the CPU utilization becomes high in the server while vulnerability scan starts and persisits.

Is there any way in which this issue can be tackled? will adding grok_pattern help here?
grok_pattern

Received an event that has a different character encoding than you configured - Logstash CPU

Badger · July 12, 2024, 2:49pm

You would have to test that to find out. Try

grok_pattern => "^"

which will always match and also be very fast. However, it will not parse anything.

If that resolves the problem then make sure that

grok_pattern =>"<%{POSINT:priority}>%{SYSLOGLINE}"

recreates the problem, then I would suggest

grok_pattern =>"^<%{POSINT:priority}>%{SYSLOGLINE}"

and then expand the SYSLOGLINE pattern and keep removing sub-patterns from the end until the problem goes away....

grok_pattern => "<%{POSINT:priority}>(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|) %{GREEDYDATA:message}"

amaljoy · July 15, 2024, 4:21am

Thank you @Badger .

I'll check this out and update you. Vulnerability scan happens only on fridays so i'll have to wait till then to give an update.

amaljoy · July 24, 2024, 3:43am

Hi @Badger,
Unfortunately the scan caused high cpu with the above grok too.
I have used the hot threads API to find out the events during vulnerability scan.

{
  "host" : "lpelka02.sysdomain.local",
  "version" : "8.5.2",
  "http_address" : "127.0.0.1:9600",
  "id" : "657d73dc-6cad-4813-bd79-148c3a097d9b",
  "name" : "lpelka02.sysdomain.local",
  "ephemeral_id" : "832aa8d6-627c-4b42-94e5-c801e2086a36",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 2,
    "batch_size" : 125,
    "batch_delay" : 5
  },
  "hot_threads" : {
    "time" : "2024-07-19T21:06:08+10:00",
    "busiest_threads" : 10,
    "threads" : [ {
      "name" : "Ruby-0-Thread-116",
      "thread_id" : 146,
      "percent_of_cpu_time" : 10.08,
      "state" : "waiting",
      "path" : ":1",
      "traces" : [ "java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)", "java.base@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)", "java.base@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938)", "java.base@17.0.5/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)", "java.base@17.0.5/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)", "app//org.logstash.ackedqueue.Queue.write(Queue.java:414)", "app//org.logstash.ackedqueue.ext.JRubyAckedQueueExt.rubyWrite(JRubyAckedQueueExt.java:129)", "app//org.logstash.ext.JrubyAckedWriteClientExt.doPush(JrubyAckedWriteClientExt.java:78)", "app//org.logstash.ext.JRubyWrappedWriteClientExt.push(JRubyWrappedWriteClientExt.java:113)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a2c00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$block$decode$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:273)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012f7000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_codec_minus_plain_minus_3_dot_1_dot_0.lib.logstash.codecs.plain.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:54)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801380800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080133b400.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:269)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801362400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c1000.linkToCallSite(LambdaForm$MH)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:196)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:185)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801277400.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "app//org.jruby.RubyClass.finvokeWithRefinements(RubyClass.java:514)", "app//org.jruby.RubyBasicObject.send(RubyBasicObject.java:1733)", "app//org.jruby.RubyKernel.send(RubyKernel.java:2243)", "app//org.jruby.RubyKernel$INVOKER$s$send.call(RubyKernel$INVOKER$s$send.gen)" ]
    }, {
      "name" : "input|syslog|tcp|172.21.90.74:58250}",
      "thread_id" : 291,
      "percent_of_cpu_time" : 7.72,
      "state" : "blocked",
      "traces" : [ "app//org.jruby.util.io.SelectorPool.get(SelectorPool.java:78)", "app//org.jruby.RubyThread.select(RubyThread.java:2124)", "app//org.jruby.RubyThread.select(RubyThread.java:2053)", "app//org.jruby.util.io.OpenFile.waitReadable(OpenFile.java:1437)", "app//org.jruby.util.io.OpenFile.fillbuf(OpenFile.java:1298)", "app//org.jruby.util.io.OpenFile.getlineFast(OpenFile.java:1635)", "app//org.jruby.RubyIO.getlineImpl(RubyIO.java:716)", "app//org.jruby.RubyIO.access$100(RubyIO.java:122)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2492)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2487)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:129)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:39)", "app//org.jruby.RubyIO.each(RubyIO.java:3533)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080129b000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a2c00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012d2400.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$tcp_receiver$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:235)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080130cc00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$block$tcp_listener$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:217)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012cf400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "[toMS]>worker1",
      "thread_id" : 99,
      "percent_of_cpu_time" : 4.89,
      "state" : "timed_waiting",
      "traces" : [ "java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@17.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)", "java.base@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1757)", "app//org.logstash.ackedqueue.Queue.readPageBatch(Queue.java:634)", "app//org.logstash.ackedqueue.Queue.readBatch(Queue.java:607)", "app//org.logstash.ackedqueue.ext.JRubyAckedQueueExt.readBatch(JRubyAckedQueueExt.java:151)", "app//org.logstash.ackedqueue.AckedReadBatch.create(AckedReadBatch.java:49)", "app//org.logstash.ext.JrubyAckedReadClientExt.readBatch(JrubyAckedReadClientExt.java:87)", "app//org.logstash.execution.WorkerLoop.run(WorkerLoop.java:82)", "java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)", "java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568)", "app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:442)", "app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:306)", "app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x00000008012a3800.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a4800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:301)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012cf400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "[toMS]>worker0",
      "thread_id" : 98,
      "percent_of_cpu_time" : 4.79,
      "state" : "runnable",
      "traces" : [ "app//org.jruby.RubyArray.hashImpl(RubyArray.java:842)", "app//org.jruby.RubyArray.hash(RubyArray.java:834)", "app//org.jruby.RubyArray$INVOKER$i$0$0$hash.call(RubyArray$INVOKER$i$0$0$hash.gen)", "app//org.jruby.runtime.Helpers.invokedynamic(Helpers.java:2617)", "app//org.jruby.RubyObject.hashCode(RubyObject.java:431)", "com.concurrent_ruby.ext.jsr166e.ConcurrentHashMapV8.internalGet(ConcurrentHashMapV8.java:1238)", "com.concurrent_ruby.ext.jsr166e.ConcurrentHashMapV8.get(ConcurrentHashMapV8.java:2676)", "com.concurrent_ruby.ext.JRubyMapBackendLibrary$JRubyMapBackend.op_aref(JRubyMapBackendLibrary.java:118)", "com.concurrent_ruby.ext.JRubyMapBackendLibrary$JRubyMapBackend$INVOKER$i$1$0$op_aref.call(JRubyMapBackendLibrary$JRubyMapBackend$INVOKER$i$1$0$op_aref.gen)", "app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:842)", "app//org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "app//org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1202)", "app//org.jruby.ir.targets.indy.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:29)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x00000008012d7400.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801305800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c1000.linkToCallSite(LambdaForm$MH)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.map.RUBY$method$\\=\\{\\}$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/map.rb:136)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012dc000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.logstash_minus_core.lib.logstash.instrument.metric_store.RUBY$method$fetch_or_store$0(/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:76)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801355000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012f4000.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c1000.linkToCallSite(LambdaForm$MH)", "usr.share.logstash.logstash_minus_core.lib.logstash.instrument.collector.RUBY$method$get$0(/usr/share/logstash/logstash-core/lib/logstash/instrument/collector.rb:65)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801362400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c2400.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c1000.linkToCallSite(LambdaForm$MH)", "usr.share.logstash.logstash_minus_core.lib.logstash.instrument.collector.RUBY$method$push$0(/usr/share/logstash/logstash-core/lib/logstash/instrument/collector.rb:50)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x0000000801c1c800.invokeStatic(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801277400.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "app//org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)", "app//org.jruby.RubyClass.finvoke(RubyClass.java:784)", "app//org.jruby.runtime.Helpers.invoke(Helpers.java:661)", "app//org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:366)" ]
    }, {
      "name" : "[main]>worker0",
      "thread_id" : 143,
      "percent_of_cpu_time" : 3.37,
      "state" : "waiting",
      "traces" : [ "java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)", "java.base@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)", "java.base@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938)", "java.base@17.0.5/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)", "java.base@17.0.5/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)", "app//org.logstash.ackedqueue.Queue.readBatch(Queue.java:605)", "app//org.logstash.ackedqueue.ext.JRubyAckedQueueExt.readBatch(JRubyAckedQueueExt.java:151)", "app//org.logstash.ackedqueue.AckedReadBatch.create(AckedReadBatch.java:49)", "app//org.logstash.ext.JrubyAckedReadClientExt.readBatch(JrubyAckedReadClientExt.java:87)", "app//org.logstash.execution.WorkerLoop.run(WorkerLoop.java:82)", "jdk.internal.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)", "java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568)", "app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:442)", "app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:306)", "app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x00000008012a3800.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a4800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:301)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012cf400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "[main]>worker1",
      "thread_id" : 144,
      "percent_of_cpu_time" : 3.22,
      "state" : "runnable",
      "traces" : [ "java.base@17.0.5/java.nio.MappedMemoryUtils.force0(Native Method)", "java.base@17.0.5/java.nio.MappedMemoryUtils.force(MappedMemoryUtils.java:100)", "java.base@17.0.5/java.nio.Buffer$1.force(Buffer.java:839)", "java.base@17.0.5/jdk.internal.misc.ScopedMemoryAccess.forceInternal(ScopedMemoryAccess.java:337)", "java.base@17.0.5/jdk.internal.misc.ScopedMemoryAccess.force(ScopedMemoryAccess.java:325)", "java.base@17.0.5/java.nio.MappedByteBuffer.force(MappedByteBuffer.java:309)", "java.base@17.0.5/java.nio.MappedByteBuffer.force(MappedByteBuffer.java:250)", "app//org.logstash.ackedqueue.io.MmapPageIOV2.ensurePersisted(MmapPageIOV2.java:218)", "app//org.logstash.ackedqueue.Page.headPageCheckpoint(Page.java:194)", "app//org.logstash.ackedqueue.Page.checkpoint(Page.java:184)", "app//org.logstash.ackedqueue.Page.ack(Page.java:164)", "app//org.logstash.ackedqueue.Queue.ack(Queue.java:717)", "app//org.logstash.ackedqueue.Batch.close(Batch.java:60)", "app//org.logstash.ackedqueue.AckedBatch.close(AckedBatch.java:56)", "app//org.logstash.ackedqueue.AckedReadBatch.close(AckedReadBatch.java:92)", "app//org.logstash.execution.QueueReadClientBase.closeBatch(QueueReadClientBase.java:132)", "app//org.logstash.execution.WorkerLoop.run(WorkerLoop.java:92)", "jdk.internal.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)", "java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568)", "app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:442)", "app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:306)", "app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x00000008012a3800.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a4800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129e800.reinvoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129ec00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:301)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012cf400.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "input|syslog|tcp|172.21.90.74:40288}",
      "thread_id" : 302,
      "percent_of_cpu_time" : 3.14,
      "state" : "blocked",
      "traces" : [ "app//org.jruby.util.io.SelectorPool.get(SelectorPool.java:78)", "app//org.jruby.RubyThread.select(RubyThread.java:2124)", "app//org.jruby.RubyThread.select(RubyThread.java:2053)", "app//org.jruby.util.io.OpenFile.readInternal(OpenFile.java:1394)", "app//org.jruby.util.io.OpenFile.fillbuf(OpenFile.java:1295)", "app//org.jruby.util.io.OpenFile.getlineFast(OpenFile.java:1635)", "app//org.jruby.RubyIO.getlineImpl(RubyIO.java:716)", "app//org.jruby.RubyIO.access$100(RubyIO.java:122)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2492)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2487)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:129)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:39)", "app//org.jruby.RubyIO.each(RubyIO.java:3533)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080129b000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a2c00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012d2400.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$tcp_receiver$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:235)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080130cc00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$block$tcp_listener$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:217)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801ff9000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "LogStash::Runner",
      "thread_id" : 1,
      "percent_of_cpu_time" : 2.97,
      "state" : "timed_waiting",
      "traces" : [ "java.base@17.0.5/java.lang.Object.wait(Native Method)", "java.base@17.0.5/java.lang.Thread.join(Thread.java:1298)", "app//org.jruby.internal.runtime.RubyNativeThread.join(RubyNativeThread.java:65)", "app//org.jruby.RubyThread.join(RubyThread.java:1244)", "app//org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)", "app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:820)", "app//org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:840)", "app//org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080127a000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129d800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.stud_minus_0_dot_0_dot_23.lib.stud.task.RUBY$method$wait$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/task.rb:37)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)", "app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)", "app//org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "app//org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:351)", "app//org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:144)", "app//org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345)", "app//org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "app//org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080127a000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080129d800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:66)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801277400.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "app//org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "app//org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1202)", "app//org.jruby.ir.targets.indy.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:29)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x00000008012d7400.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801305800.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012c1000.linkToCallSite(LambdaForm$MH)", "usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:282)", "usr.share.logstash.logstash_minus_core.lib.logstash.runner.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:280)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801277400.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "app//org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080127a000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080127ac00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)" ]
    }, {
      "name" : "input|syslog|tcp|172.21.90.74:49246}",
      "thread_id" : 308,
      "percent_of_cpu_time" : 2.62,
      "state" : "blocked",
      "traces" : [ "app//org.jruby.util.io.SelectorPool.get(SelectorPool.java:78)", "app//org.jruby.RubyThread.select(RubyThread.java:2124)", "app//org.jruby.RubyThread.select(RubyThread.java:2053)", "app//org.jruby.util.io.OpenFile.waitReadable(OpenFile.java:1437)", "app//org.jruby.util.io.OpenFile.fillbuf(OpenFile.java:1298)", "app//org.jruby.util.io.OpenFile.getlineFast(OpenFile.java:1635)", "app//org.jruby.RubyIO.getlineImpl(RubyIO.java:716)", "app//org.jruby.RubyIO.access$100(RubyIO.java:122)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2492)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2487)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:129)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:39)", "app//org.jruby.RubyIO.each(RubyIO.java:3533)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080129b000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a2c00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012d2400.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$tcp_receiver$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:235)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080130cc00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$block$tcp_listener$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:217)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801ff9000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    }, {
      "name" : "input|syslog|tcp|172.21.90.74:51204}",
      "thread_id" : 311,
      "percent_of_cpu_time" : 2.45,
      "state" : "blocked",
      "traces" : [ "app//org.jruby.util.io.SelectorPool.get(SelectorPool.java:78)", "app//org.jruby.RubyThread.select(RubyThread.java:2124)", "app//org.jruby.RubyThread.select(RubyThread.java:2053)", "app//org.jruby.util.io.OpenFile.waitReadable(OpenFile.java:1437)", "app//org.jruby.util.io.OpenFile.fillbuf(OpenFile.java:1298)", "app//org.jruby.util.io.OpenFile.getlineFast(OpenFile.java:1635)", "app//org.jruby.RubyIO.getlineImpl(RubyIO.java:716)", "app//org.jruby.RubyIO.access$100(RubyIO.java:122)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2492)", "app//org.jruby.RubyIO$4.getline(RubyIO.java:2487)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:129)", "app//org.jruby.util.io.Getline.getlineCall(Getline.java:39)", "app//org.jruby.RubyIO.each(RubyIO.java:3533)", "java.base@17.0.5/java.lang.invoke.LambdaForm$DMH/0x000000080129b000.invokeVirtual(LambdaForm$DMH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012a2c00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012d2400.tryFinally(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801281c00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$method$tcp_receiver$0(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:235)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x000000080130cc00.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x00000008012bfc00.guard(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_input_minus_syslog_minus_3_dot_6_dot_0.lib.logstash.inputs.syslog.RUBY$block$tcp_listener$1(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-syslog-3.6.0/lib/logstash/inputs/syslog.rb:217)", "java.base@17.0.5/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000801ff9000.invoke(LambdaForm$MH)", "java.base@17.0.5/java.lang.invoke.LambdaForm$MH/0x0000000800cad000.invokeExact_MT(LambdaForm$MH)", "app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "app//org.jruby.runtime.Block.call(Block.java:143)", "app//org.jruby.RubyProc.call(RubyProc.java:309)", "app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base@17.0.5/java.lang.Thread.run(Thread.java:833)" ]
    } ]
  }
}

amaljoy · August 12, 2024, 6:44am

Finally this issue has been resolved.

Changes i have made:

listening on the logstash ip rather than 0.0.0.0.
made changes in pipelines config to drop the vulnerable scanner messages.
changed codec.
In jvm.options increased the intial and max heap size to 8GB. (32GB linux server)

input {
 syslog {
  id => "idsyslog"
  host => "172.29.144.38"
  port => 10514
  type => "syslog"
  codec => plain {
                    charset => "ISO-8859-1"
            }
 }
}
filter {
 if [host][ip] == "(vulnerable scanner ip)" {
 drop{}
 }

Thank you @leandrojmp @Badger

Topic		Replies	Views
CPU usage for logstash hits over 300% Logstash	6	1113	December 23, 2020
High CPU on logstash cluster Logstash	5	867	August 22, 2017
Logstash 7.12.0 High CPU usage Logstash	6	3020	May 10, 2021
Logstash performance drop with high CPU usage Logstash	2	935	July 6, 2017
Logstash CPU gets pegged, all processing stops with Apache logs Logstash	11	711	January 24, 2022

Logstash Vulnerability scanner, High CPU utilization

Related topics