Hey, so i had written an input configuration for reading GC log files(which are actually on size based rolling) and used multiline codec with auto_flush_interval set to 1.I am using logstash version 6.1.1. It works and everything but eventually logstash becomes unresponsive. It stops sending any data to elasticsearch. I have to restart it for it to work again. I took a thread dump of logstash in it's unresponsive state. There were 2 threads in blocking state. Here's the thread dump -
Ruby-0-Thread-35: path/to/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/filewatch-0.9.0/lib/filewatch/watch.rb:176 - priority:5 - threadId:0x00007f57fc01a800 - nativeId:0x1edc - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:47)
- waiting to lock <0x00000000ddbb0110> (a org.jruby.RubyHash)
at java.lang.invoke.LambdaForm$DMH/1858609436.invokeVirtual_L5_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/22069592.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1957502751.linkToCallSite(LambdaForm$MH)
at CLPPRDWEB014.home.webprd01.tmon.tmon_minus_agent_minus_3_dot_4_dot_1.logstash_minus_6_dot_1_dot_1.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_codec_minus_multiline_minus_3_dot_0_dot_8.lib.logstash.codecs.identity_map_codec.RUBY$method$identity_count$0(/CLPPRDWEB014/home/webprd01/tmon/tmon-agent-3.4.1/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.8/lib/logstash/codecs/identity_map_codec.rb:241)
at java.lang.invoke.LambdaForm$DMH/2061475679.invokeStatic_L6_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/1485697819.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1957502751.linkToCallSite(LambdaForm$MH)
at CLPPRDWEB014.home.webprd01.tmon.tmon_minus_agent_minus_3_dot_4_dot_1.logstash_minus_6_dot_1_dot_1.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_codec_minus_multiline_minus_3_dot_0_dot_8.lib.logstash.codecs.identity_map_codec.RUBY$method$map_cleanup$0(/CLPPRDWEB014/home/webprd01/tmon/tmon-agent-3.4.1/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.8/lib/logstash/codecs/identity_map_codec.rb:249)
at java.lang.invoke.LambdaForm$DMH/92523742.invokeStatic_L6_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1027007693.invokeExact_MT(LambdaForm$MH)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:90)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
at org.jruby.RubyClass.finvoke(RubyClass.java:512)
at org.jruby.RubyBasicObject.send19(RubyBasicObject.java:1664)
at org.jruby.RubyKernel.send19(RubyKernel.java:1982)
at java.lang.invoke.LambdaForm$DMH/1247233941.invokeStatic_L4_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/209833425.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/603742814.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1154002927.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/603742814.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1154002927.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1615056168.linkToCallSite(LambdaForm$MH)
at CLPPRDWEB014.home.webprd01.tmon.tmon_minus_agent_minus_3_dot_4_dot_1.logstash_minus_6_dot_1_dot_1.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_codec_minus_multiline_minus_3_dot_0_dot_8.lib.logstash.codecs.identity_map_codec.RUBY$block$start$1(/CLPPRDWEB014/home/webprd01/tmon/tmon-agent-3.4.1/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-codec-multiline-3.0.8/lib/logstash/codecs/identity_map_codec.rb:55)
at java.lang.invoke.LambdaForm$DMH/1804094807.invokeStatic_L8_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1095293768.invokeExact_MT(LambdaForm$MH)
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
at org.jruby.runtime.Block.call(Block.java:124)
at org.jruby.RubyProc.call(RubyProc.java:289)
at org.jruby.RubyProc.call(RubyProc.java:246)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
at java.lang.Thread.run(Thread.java:745)Ruby-0-Thread-33: path/to/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/filewatch-0.9.0/lib/filewatch/observing_tail.rb:79 - priority:5 - threadId:0x00007f57f4056000 - nativeId:0x1eda - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:47)
- waiting to lock <0x00000000de51a1c0> (a org.jruby.RubyHash)
at java.lang.invoke.LambdaForm$DMH/1858609436.invokeVirtual_L5_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/22069592.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1160460865.delegate(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1725017993.guard(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1957502751.linkToCallSite(LambdaForm$MH)
.....
Here's my logstash configuration -
input {
file {
path => "path/to/logs/gc*"
sincedb_path => "path/to/gc.db"
start_position => "end"
type => "GC_Data"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}:"
negate => true
what => "previous"
auto_flush_interval => 1
}
}
}
Could you please help me out in this? I can't seem to pinpoint the issue. Is it an error in the plugin?