Logstash - undefined method 'to_msgpack' for 0.2e2:BigDecimal

Hey all,

Recently have been doing some JSON parsing of messages and it appears that every once in awhile, we come across a message that just breaks logstash and crashes it with a fatal error. I am using the s3-sns-sqs plugin for input. It appears this error relates to my json parsing within the filter section which I will paste below:

filter {
mutate {
add_field => {
"key" => "%{[@metadata][s3][object_key]}"
}
}

json {
    source => "message"
    target => "message"
    skip_on_invalid_json => "true"
}

json {
    source => "[message][cont_meta_data]"
    target => "[message][cont_meta_data]"
    skip_on_invalid_json => "true"
}

}

Error stack trace:

FATAL] 2019-08-12 00:01:58.584 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `to_msgpack' for 0.2e2:BigDecimal, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:239)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[ERROR] 2019-08-12 00:01:58.603 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Is there a good way to capture an error like the above and tell LogStash to ignore and continue processing the remaining files? Would a dead letter queue be of some use here to catch any errors like the above instead of halting the process entirely?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.