# Skip\_on\_invalid\_json skipping all filters

**URL:** <https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195>\
**Category:** Logstash\
**Created:** [January 15, 2020, 7:58pm UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195 "2020-01-15T19:58:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jfcantu/32/60853_2.png) [@jfcantu](https://discuss.elastic.co/u/jfcantu)\
**Post date:** [January 15, 2020, 7:58pm UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195/1 "2020-01-15T19:58:53Z")

</div>

Hi,

I ran across something unexpected yesterday, that doesn't seem to line up with the Logstash documentation, and I was wondering if this was intended behavior (or if I did something wrong.)

TL;DR - if you have multiple `filter {}` blocks, and attempt to use `json {}` in one of the filters with `skip_on_invalid_json` set, it seems to skip _all_ filters rather than just the filter where `json{}` was used.

* * *

Longer version:

Per the [JSON filter documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-skip_on_invalid_json) on `skip_on_invalid_json` (emphasis added):

> Allows for skipping **the filter** on invalid JSON.

Based on this, I assumed that if `json {}` hits invalid JSON, it will skip the rest of the parent filter - but still process other filters in the pipeline.

So, I had the following in my pipeline definition:

```
filter {
  json {
    skip_on_invalid_json => true
    source => "message"
  }
  
  mutate {
    remove_field => "message"
  }
}

filter {
  mutate {
    # do some other stuff
  }
}

```

What I found was, when [message] didn't contain valid JSON, the second filter didn't seem to run at all. I added a "JSON check" prior to the `json {}` block by putting `if [message] =~ "^\s*{.*}\s*$" { }` around it, and that seemed to solve the problem - if [message] doesn't look like a JSON object, the `json {}` block doesn't run, `skip_on_invalid_json` is never encountered, and the second filter runs as expected.

Is this expected behavior?

---

<div class="post-metadata">

**Author:** ![logger](https://avatars.discourse-cdn.com/v4/letter/l/34f0e0/32.png) [@logger](https://discuss.elastic.co/u/logger)\
**Post date:** [January 15, 2020, 9:15pm UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195/2 "2020-01-15T21:15:54Z")

</div>

Hi,

it does NOT skip the parent "filter {}" block. it just skips trying to execute the json{} filter.  
So if it is no valid json it just removes the "message" field and after that it tries the "second" filter block. BUT without the message field.

you can check this by using stdout or file output.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 15, 2020, 10:24pm UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195/3 "2020-01-15T22:24:15Z")

</div>

Do the fields that the later mutate works on exist if the JSON is invalid?

Instead of unconditionally removing [message] with a mutate I would add

```
 remove_field => ["message"]

```

to the json filter, so that it is only removed if it is successfully parsed.

---

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jfcantu/32/60853_2.png) [@jfcantu](https://discuss.elastic.co/u/jfcantu)\
**Post date:** [January 16, 2020, 12:06am UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195/4 "2020-01-16T00:06:54Z")

</div>

Thank you both - it turned out to be two things I didn't know:

- Exactly what you said - `mutate { remove_field => ["message"] }` was still running even after the JSON parse failure.
- The `mutate {}` in the second filter was trying to remove a protected field (`@version`), which I didn't realize you can't do. I was still seeing `@version` in ElasticSearch and I assumed the filter wasn't running at all.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 13, 2020, 12:06am UTC](https://discuss.elastic.co/t/skip-on-invalid-json-skipping-all-filters/215195/5 "2020-02-13T00:06:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.