Crash of logstash through KV plugin

Elastic Stack Logstash
1

I'm running logstash 6.1.3 and have been intermittently getting an Array index out of bounds. Nothing turns up in the logs, and as such I'm not able to provide exact data that can replicate it. Also, its intermittent. I spent a long time tonight trying to replicate it by replaying parts of our data stream, to no avail. I'd move through a large sample, hit the error, re-run on the smaller subset, and not hit the error again. It appears to be coming from the KV filter, which logically makes some sense, as the data we are working with is really dirty and malformed.

Looking at the logstash-filter-kv code, I might get more insights at the debug log level, so I have now switched this process to use that. Hopefully I will have some more insight in the morning.

Here is as much of the exception and stack trace as I could pull off the console.

IndexOutOfBoundsException: -1
        at java.util.ArrayList.elementData(ArrayList.java:422)
        at java.util.ArrayList.remove(ArrayList.java:499)
        at org.logstash.FieldReference.parse(FieldReference.java:165)
        at org.logstash.FieldReference.parseToCache(FieldReference.java:152)
        at org.logstash.FieldReference.from(FieldReference.java:84)
        at org.logstash.ConvertedMap.convertKey(ConvertedMap.java:101)
        at org.logstash.ConvertedMap.access$000(ConvertedMap.java:23)
        at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:34)
        at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:28)
        at org.jruby.RubyHash.visitLimited(RubyHash.java:662)
        at org.jruby.RubyHash.visitAll(RubyHash.java:647)
        at org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:68)
        at org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:63)
        at org.logstash.Valuefier.lambda$initConverters$11(Valuefier.java:142)
        at org.logstash.Valuefier.convert(Valuefier.java:73)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:95)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i
$2$0$ruby_set_field.gen)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_kv_minus_4_dot_1_dot_0.lib.logstash
.filters.kv.invokeOther29:set(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-kv-4.1.0/lib/logstash/filters/kv.rb:
397)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_kv_minus_4_dot_1_dot_0.lib.logstash
.filters.kv.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-kv-4.1.0/lib/logstash/filters/kv.
rb:397)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib
/logstash/filters/base.rb:145)
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core
/lib/logstash/filters/base.rb:145)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(/usr/share/logstash/logstash-core/
lib/logstash/filters/base.rb:164)
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-co
re/lib/logstash/filters/base.rb:164)
        at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
        at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
        at org.jruby.runtime.Block.yield(Block.java:165)
        at org.jruby.RubyArray.each(RubyArray.java:1734)
        at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)
2

I can now fully reproduce this. I'll be submitting a bug report on github. Submitted as https://github.com/logstash-plugins/logstash-filter-kv/issues/56

3

So, no comments here. None on the github issue. I guess no one cares about malformed input crashing the KV plugin.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.