Skip_on_invalid_json skipping all filters

[jfcantu]

Hi,

I ran across something unexpected yesterday, that doesn't seem to line up with the Logstash documentation, and I was wondering if this was intended behavior (or if I did something wrong.)

TL;DR - if you have multiple filter {} blocks, and attempt to use json {} in one of the filters with skip_on_invalid_json set, it seems to skip all filters rather than just the filter where json{} was used.

---

Longer version:

Per the JSON filter documentation on skip_on_invalid_json (emphasis added):

Allows for skipping the filter on invalid JSON.

Based on this, I assumed that if json {} hits invalid JSON, it will skip the rest of the parent filter - but still process other filters in the pipeline.

So, I had the following in my pipeline definition:

filter {
  json {
    skip_on_invalid_json => true
    source => "message"
  }
  
  mutate {
    remove_field => "message"
  }
}

filter {
  mutate {
    # do some other stuff
  }
}

What I found was, when [message] didn't contain valid JSON, the second filter didn't seem to run at all. I added a "JSON check" prior to the json {} block by putting if [message] =~ "^\s*{.*}\s*$" { } around it, and that seemed to solve the problem - if [message] doesn't look like a JSON object, the json {} block doesn't run, skip_on_invalid_json is never encountered, and the second filter runs as expected.

Is this expected behavior?

[logger (Malte Leek)]

Hi,

it does NOT skip the parent "filter {}" block. it just skips trying to execute the json{} filter.
So if it is no valid json it just removes the "message" field and after that it tries the "second" filter block. BUT without the message field.

you can check this by using stdout or file output.

[Badger]

Do the fields that the later mutate works on exist if the JSON is invalid?

Instead of unconditionally removing [message] with a mutate I would add

 remove_field => [ "message" ]

to the json filter, so that it is only removed if it is successfully parsed.

[jfcantu]

Thank you both - it turned out to be two things I didn't know:

• Exactly what you said - mutate { remove_field => ["message"] } was still running even after the JSON parse failure.
• The mutate {} in the second filter was trying to remove a protected field (@version), which I didn't realize you can't do. I was still seeing @version in ElasticSearch and I assumed the filter wasn't running at all.