Logstash stopped transmitting events

[dandrestor (Dan Nestor)]

Hello,

I need help troubleshooting a problem that occurred Saturday night, with no change to the configuration: Logstash stopped emitting events to Elasticsearch. There is nothing in the logs that I can see to indicate any problem. I restarted the Logstash daemon, as well as tried different output configurations, to no effect.

Logstash is configured with a persistent queue, and new logs are being accumulated in it, but are apparently not processed further. The only messages that are sent further to Elasticsearch are the self-monitoring events generated by Logstash itself.

The Logstash version is 5.5.2, running on Linux. The configuration is nothing special, if rather verbose - a few inputs and some filters, with a single output to Elasticsearch. Again, the problem started during the night with no change to the config, which makes me suspect an environmental cause.

Thanks for your thoughts!
//Dan

[dandrestor (Dan Nestor)]

I think I found the error message:

Oct  9 10:48:12 x0181se logstash: Exception in thread "[main]>worker0" org.logstash.ackedqueue.QueueRuntimeException: deserialize invocation error
Oct  9 10:48:12 x0181se logstash: Caused by: java.lang.reflect.InvocationTargetException
Oct  9 10:48:12 x0181se logstash: Caused by: java.lang.IllegalArgumentException: Missing Valuefier handling for full class name=[B, simple name=byte[]
Oct  9 10:48:12 x0181se logstash: Caused by: java.lang.IllegalArgumentException: No enum constant org.logstash.bivalues.BiValues.[B

Looks like I'm hitting https://github.com/elastic/logstash/issues/8379

Is there any workaround that will allow Logstash to continue processing the events in the queue? Right now I'm expecting my queue to fill up pretty soon at which point data loss will occur.