I'm running logstash 6.1.3 and have been intermittently getting an Array index out of bounds. Nothing turns up in the logs, and as such I'm not able to provide exact data that can replicate it. Also, its intermittent. I spent a long time tonight trying to replicate it by replaying parts of our data stream, to no avail. I'd move through a large sample, hit the error, re-run on the smaller subset, and not hit the error again. It appears to be coming from the KV filter, which logically makes some sense, as the data we are working with is really dirty and malformed.
Looking at the logstash-filter-kv code, I might get more insights at the debug log level, so I have now switched this process to use that. Hopefully I will have some more insight in the morning.
Here is as much of the exception and stack trace as I could pull off the console.
IndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:422)
at java.util.ArrayList.remove(ArrayList.java:499)
at org.logstash.FieldReference.parse(FieldReference.java:165)
at org.logstash.FieldReference.parseToCache(FieldReference.java:152)
at org.logstash.FieldReference.from(FieldReference.java:84)
at org.logstash.ConvertedMap.convertKey(ConvertedMap.java:101)
at org.logstash.ConvertedMap.access$000(ConvertedMap.java:23)
at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:34)
at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:28)
at org.jruby.RubyHash.visitLimited(RubyHash.java:662)
at org.jruby.RubyHash.visitAll(RubyHash.java:647)
at org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:68)
at org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:63)
at org.logstash.Valuefier.lambda$initConverters$11(Valuefier.java:142)
at org.logstash.Valuefier.convert(Valuefier.java:73)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:95)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i
$2$0$ruby_set_field.gen)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_kv_minus_4_dot_1_dot_0.lib.logstash
.filters.kv.invokeOther29:set(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-kv-4.1.0/lib/logstash/filters/kv.rb:
397)
at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_kv_minus_4_dot_1_dot_0.lib.logstash
.filters.kv.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-kv-4.1.0/lib/logstash/filters/kv.
rb:397)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib
/logstash/filters/base.rb:145)
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core
/lib/logstash/filters/base.rb:145)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(/usr/share/logstash/logstash-core/
lib/logstash/filters/base.rb:164)
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-co
re/lib/logstash/filters/base.rb:164)
at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
at org.jruby.runtime.Block.yield(Block.java:165)
at org.jruby.RubyArray.each(RubyArray.java:1734)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)