How to Handle Exceptions?

Elastic Stack Logstash
1

It seems that if logstash encounters an exception in the elasticsearch output pipeline it will completely halt processing of any further entries.

[2019-08-08T22:14:44,075][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.ClassCastException: java.util.HashMap cannot be cast to java.lang.String
        at org.logstash.Event.existingTag(Event.java:398) ~[logstash-core.jar:?]
        at org.logstash.Event.tag(Event.java:374) ~[logstash-core.jar:?]
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_tag(JrubyEventExtLibrary.java:257) ~[logstash-core.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_1_dot_8_minus_java.lib.logstash.inputs.beats.raw_event_transform.RUBY$method$transform$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstas
h-input-beats-5.1.8-java/lib/logstash/inputs/beats/raw_event_transform.rb:14) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_1_dot_8_minus_java.lib.logstash.inputs.beats.message_listener.RUBY$method$onNewMessage$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstas
h-input-beats-5.1.8-java/lib/logstash/inputs/beats/message_listener.rb:41) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_1_dot_8_minus_java.lib.logstash.inputs.beats.message_listener.RUBY$method$onNewMessage$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/
gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats/message_listener.rb) ~[?:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) ~[jruby-complete-9.2.6.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) ~[jruby-complete-9.2.6.0.jar:?]
        at LogStash$$Inputs$$Beats$$MessageListener_1219897284.onNewMessage(LogStash$$Inputs$$Beats$$MessageListener_1219897284.gen:13) ~[?:?]
        at org.logstash.beats.BeatsHandler.channelRead0(BeatsHandler.java:52) ~[logstash-input-beats-5.1.8.jar:?]
        at org.logstash.beats.BeatsHandler.channelRead0(BeatsHandler.java:12) ~[logstash-input-beats-5.1.8.jar:?]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

I've tried looking up documentation on logstash configuration files to see if there is any way to handle this scenario, but I have found nothing.

So I ask here: how are you supposed to configure logstash to handle exceptions, or are logstash configuration files not the place to do it?

2

It's up to the author of the code to handle exceptions. As an end-user you cannot do much. The exception is in the beats input.

That said, does setting the option 'include_codec_tag => false' on the beats input help?

3

What does 'include_codec_tag' do? The documentation for it appears to be blank.

4

Adds a tag to the event. Disabling it may work around the bug, or it may just move the place where the bug blows things up.

5

That worked! Thank you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.