Logstash thread using high CPU usage

Elastic Stack Logstash
1

Hi,

Following threads in our logstash is using high CPU usage

======================
bash-4.2$ curl -XGET 'localhost:9600/_node/hot_threads?human=true'
::: {}
Hot threads at 2022-02-18T19:50:45+01:00, busiestThreads=10:

90.56 % of cpu usage, state: runnable, thread name: '[main]>worker7', thread id: 42
app//org.joni.ByteCodeMachine.execute(ByteCodeMachine.java:185)
app//org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:167)
app//org.joni.Matcher.matchCheck(Matcher.java:287)
app//org.joni.Matcher.searchCommon(Matcher.java:460)
app//org.joni.Matcher.search(Matcher.java:301)
app//org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:231)
app//org.jruby.RubyRegexp.search(RubyRegexp.java:1331)
app//org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1220)
app//org.jruby.RubyRegexp.matchCommon(RubyRegexp.java:1188)

90.4 % of cpu usage, state: runnable, thread name: '[main]>worker3', thread id: 38
app//org.jcodings.Encoding.isMbcWord(Encoding.java:487)
app//org.joni.ByteCodeMachine.opWordBound(ByteCodeMachine.java:1132)
app//org.joni.ByteCodeMachine.execute(ByteCodeMachine.java:218)
app//org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:167)
app//org.joni.Matcher.matchCheck(Matcher.java:287)
app//org.joni.Matcher.searchCommon(Matcher.java:460)
app//org.joni.Matcher.search(Matcher.java:301)
app//org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:231)
app//org.jruby.RubyRegexp.search(RubyRegexp.java:1331)
app//org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1220)
app//org.jruby.RubyRegexp.matchCommon(RubyRegexp.java:1188)
app//org.jruby.RubyRegexp.match_m(RubyRegexp.java:1154)
app//org.jruby.RubyRegexp$INVOKER$i$match_m.call(RubyRegexp$INVOKER$i$match_m.gen)
app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
app//org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172)

90.37 % of cpu usage, state: runnable, thread name: '[main]>worker6', thread id: 41
app//org.joni.ByteCodeMachine.execute(ByteCodeMachine.java:185)
app//org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:167)
app//org.joni.Matcher.matchCheck(Matcher.java:287)
app//org.joni.Matcher.searchCommon(Matcher.java:460)
app//org.joni.Matcher.search(Matcher.java:301)
app//org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:231)
app//org.jruby.RubyRegexp.search(RubyRegexp.java:1331)
app//org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1220)
app//org.jruby.RubyRegexp.matchCommon(RubyRegexp.java:1188)
app//org.jruby.RubyRegexp.match_m(RubyRegexp.java:1154)
app//org.jruby.RubyRegexp$INVOKER$i$match_m.call(RubyRegexp$INVOKER$i$match_m.gen)
app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
app//org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172)

90.31 % of cpu usage, state: runnable, thread name: '[main]>worker0', thread id: 35
app//org.joni.ByteCodeMachine.execute(ByteCodeMachine.java:185)
app//org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:167)
app//org.joni.Matcher.matchCheck(Matcher.java:287)
app//org.joni.Matcher.searchCommon(Matcher.java:460)
app//org.joni.Matcher.search(Matcher.java:301)
app//org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:231)
app//org.jruby.RubyRegexp.search(RubyRegexp.java:1331)
app//org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1220)
app//org.jruby.RubyRegexp.matchCommon(RubyRegexp.java:1188)
app//org.jruby.RubyRegexp.match_m(RubyRegexp.java:1154)
app//org.jruby.RubyRegexp$INVOKER$i$match_m.call(RubyRegexp$INVOKER$i$match_m.gen)
app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:417)
app//org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jls_minus_grok_minus_0_dot_11_dot_5.lib.grok_minus_pure.invokeOther1:match(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:182)
usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.jls_minus_grok_minus_0_dot_11_dot_5.lib.grok_minus_pure.RUBY$method$execute$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:182)

Does anyone has any idea regarding this threads. Please help us here.

Thanks in advance.

2

You have some very expensive grok patterns there. There is advice about anchoring and using tiered patterns to improve performance here.

If you want more specific advice then post your grok configuration (including any custom patterns) and an example of your data.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.