Hi!
We suddenly noticed that we were no longer receiving events in Kibana/Elasticsearch, and upon further investigation we saw that the server running Logstash was using 100 % CPU.
We've tried restarting both Logstash, and the server, but we see that the persistence queue has been filled up, and that for some reason, as soon as Logstash starts up and is done reading the pages in the queue, the 4 workers we have running for our pipeline suddenly jump to ~100 % CPU usage, and are not processing the events in the queue, or any other events for that matter.
How can we go about troubleshooting what's causing the issue, besides it having something to do with the match filter? Like, what actual event is it trying to process that's causing it to halt completely? Setting "log.level: debug" didn't really provide any new insights into what's causing the problem. Neither did setting slowlogs, since nothing is showing up there either.
We're a bit stuck, to be honest.
root@elk-l-001:~# curl -XGET 'localhost:9600/_node/hot_threads?human=true&threads=5'
::: {}
Hot threads at 2019-01-10T04:04:25+01:00, busiestThreads=5:
================================================================================
82.49 % of cpu usage, state: runnable, thread name: 'Ruby-0-Thread-28', thread id: 53
:1
org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:228)
org.joni.Matcher.matchCheck(Matcher.java:304)
org.joni.Matcher.searchInterruptible(Matcher.java:469)
--------------------------------------------------------------------------------
82.41 % of cpu usage, state: runnable, thread name: 'Ruby-0-Thread-25', thread id: 50
:1
org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:228)
org.joni.Matcher.matchCheck(Matcher.java:304)
org.joni.Matcher.searchInterruptible(Matcher.java:469)
--------------------------------------------------------------------------------
82.25 % of cpu usage, state: runnable, thread name: 'Ruby-0-Thread-27', thread id: 52
:1
org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:272)
org.joni.Matcher.matchCheck(Matcher.java:304)
org.joni.Matcher.searchInterruptible(Matcher.java:469)
--------------------------------------------------------------------------------
81.57 % of cpu usage, state: runnable, thread name: 'Ruby-0-Thread-26', thread id: 51
:1
org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:272)
org.joni.Matcher.matchCheck(Matcher.java:304)
org.joni.Matcher.searchInterruptible(Matcher.java:469)
--------------------------------------------------------------------------------
0.61 % of cpu usage, state: timed_waiting, thread name: 'LogStash::Runner', thread id: 1
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1260)
org.jruby.internal.runtime.NativeThread.join(NativeThread.java:76)
--------------------------------------------------------------------------------