Logstash crashes when converting fields to ECS Format

olatunde.tokun · January 22, 2019, 10:32pm

Got some logstash errors while converting fields to the Elastic Common Schema (ECS).

Logstash version: logstash 6.0.0

The goal is to rename a source field into an ecs compliant field e.g [destination][field]

echo '{"source":"foo", "destination":"bar"}' | /usr/share/logstash/bin/logstash -e 'input {stdin{codec => json_lines}} filter {mutate {rename => {"source" => "[destination][file]"}}} output{stdout{codec => rubydebug}}'

Running this returns the errors.
Please pardon me for the ugly trace below. Any help or pointers is welcome. Thanks!

[ERROR] 2019-01-22 17:12:52.281 [[main]>worker8] pipeline - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"org.logstash.bivalues.StringBiValue cannot be cast to org.logstash.ConvertedList", "backtrace"=>["org.logstash.Accessors.setChild(Accessors.java:109)", "org.logstash.Accessors.set(Accessors.java:18)", "org.logstash.Event.setField(Event.java:161)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:132)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyHash$12.visit(RubyHash.java:1362)", "org.jruby.RubyHash$12.visit(RubyHash.java:1359)", "org.jruby.RubyHash.visitLimited(RubyHash.java:662)", "org.jruby.RubyHash.visitAll(RubyHash.java:647)", "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1319)", "org.jruby.RubyHash.each_pairCommon(RubyHash.java:1354)", "org.jruby.RubyHash.each(RubyHash.java:1343)", "org.jruby.RubyHash$INVOKER$i$0$0$each.call(RubyHash$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x511e61bc@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 sleep>"} Exception in thread "[main]>worker8" java.lang.ClassCastException: org.logstash.bivalues.StringBiValue cannot be cast to org.logstash.ConvertedList at org.logstash.Accessors.setChild(Accessors.java:109) at org.logstash.Accessors.set(Accessors.java:18) at org.logstash.Event.setField(Event.java:161) at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:132)

Badger · January 22, 2019, 10:46pm

In more recent version you get a different error message

[ERROR] 2019-01-22 17:40:57.190 [Ruby-0-Thread-4: :1] pipeline - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"Could not set field 'file' on object 'bar' to value 'foo'.This is probably due to trying to set a field like [foo][bar] = someValuewhen [foo] is not either a map or a string", "backtrace"=>["org.logstash.Accessors.setChild(Accessors.java:120)"

Either way, I would not expect it to work. "destination" is a string and you are trying to make it an object.

Badger · January 22, 2019, 11:47pm

Looking at the code , it would be nice to have a space in someValuewhen. The bigger (though still pretty small) issue I see is that it appears to be printing the field value rather than the field name: "on object 'bar'" should be "on object 'destination'". If anyone who understands the code concurs that is a bug I can file it on github.

Lastly "is not either a map or a string" doesn't look right to me. "is not either a map or a list" perhaps?

olatunde.tokun · January 23, 2019, 2:30pm

Your comments made me figure out the reasons for these errors.

I was able to get some more detail as it affects ecs conversion here

system · February 20, 2019, 2:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.