Logstash - undefined method 'to_msgpack' for 0.2e2:BigDecimal

Elastic Stack Logstash
1

Hey all,

Recently have been doing some JSON parsing of messages and it appears that every once in awhile, we come across a message that just breaks logstash and crashes it with a fatal error. I am using the s3-sns-sqs plugin for input. It appears this error relates to my json parsing within the filter section which I will paste below:

filter {
mutate {
add_field => {
"key" => "%{[@metadata][s3][object_key]}"
}
}

json {
    source => "message"
    target => "message"
    skip_on_invalid_json => "true"
}

json {
    source => "[message][cont_meta_data]"
    target => "[message][cont_meta_data]"
    skip_on_invalid_json => "true"
}

}

Error stack trace:

FATAL] 2019-08-12 00:01:58.584 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `to_msgpack' for 0.2e2:BigDecimal, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:239)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[ERROR] 2019-08-12 00:01:58.603 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Is there a good way to capture an error like the above and tell LogStash to ignore and continue processing the remaining files? Would a dead letter queue be of some use here to catch any errors like the above instead of halting the process entirely?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.