# Logstash filter not working, when all the filters are applied at once, but working when applied only one filter

**URL:** <https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186>\
**Category:** Logstash\
**Created:** [July 20, 2022, 5:16pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186 "2022-07-20T17:16:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![djrshn2346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djrshn2346/32/108594_2.png) [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\
**Post date:** [July 20, 2022, 5:16pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/1 "2022-07-20T17:16:56Z")

</div>

I have added 3 filters in Logstash but at a time only 2 are working, but all the three are not working at the same time. One of the filter is throwing error. When I applied single filter that is working fine, when 2 that is also working fine but when all are applied, one of the filter is not working.

Below is my filter configuration:

```auto
else if [type] == "tls_log" {
        json {
            source => "message"
            target => "json"
            remove_field=>["message"]
            }
        mutate {
            rename => {
              "path" => "filename"
            }
            }
        if "_jsonparsefailure" in [tags] {
          mutate {
            add_field => {
              "checker" => "value_tls_parseerror"
              "logplane" => {{ .Values.log.logplane.default | quote }}
            }
            remove_field => ["json"]
          }
        }
        else {
          mutate {
            add_field => {
              "checker" => "value_tls"
              "service_id" => "%{[json][service_id]}"
              "version" => "%{[json][version]}"
              "[metadata][container_name]" => "%{[json][metadata][container_name]}"
              "[metadata][node_name]" => "%{[json][metadata][node_name]}"
              "[metadata][namespace]" => "%{[json][metadata][namespace]}"
              "[metadata][pod_name]" => "%{[json][metadata][pod_name]}"
              "[metadata][pod_uid]" => "%{[json][metadata][pod_uid]}"
              "logplane" => {{ .Values.log.logplane.default | quote }}
              "severity" => "%{[json][severity]}"
              "message" => "%{[json][message]}"
              "timestamp" => "%{[json][timestamp]}"
            }
            remove_field => ["json"]
          }
        }
      }
      else if [type] == "metrics_log" {
        json {
            source => "message"
            target => "json"
            remove_field=>["message"]
            }
        mutate {
            rename => {
              "path" => "filename"
            }
            
            add_field => {
              "checker" => "value_metric"
              "service_id" => "%{[json][service_id]}"
              "version" => "%{[json][version]}"
              "timestamp" => "%{[json][timestamp]}"
              "[metadata][container_name]" => "%{[json][metadata][container_name]}"
              "logplane" => {{ .Values.log.logplane.default | quote }}
              "severity" => "%{[json][severity]}"
              "message" => "%{[json][message]}"
            }
            remove_field => ["json"]
          }
      }
      else if [type] == "log" {
        json {
            source => "message"
            target => "json"
            remove_field=>["message"]
            }

            if [json][facility] {
            mutate {
              add_field => { "facility" => "%{[json][facility]}" }
            }
            }

            if [json][metadata][proc_id] {
            mutate {
              add_field => { "proc_id" => "%{[json][metadata][proc_id]}" }
            }
            }

            if [json][metadata][category] {
            mutate {
              add_field => { "category" => "%{[json][metadata][category]}" }
            }
            }

        mutate {
            rename => {
              "path" => "filename"
            }
            
            add_field => {
              "checker" => "value"
              "logplane" => {{ .Values.log.logplane.default | quote }}
              "version" => "%{[json][version]}"
              "severity" => "%{[json][severity]}"
              "service_id" => "%{[json][service_id]}"
              "[kubernetes][pod][name]" => "%{[json][metadata][pod_name]}"
              "[kubernetes][pod][uid]" => "%{[json][metadata][pod_uid]}"
              "[kubernetes][namespace]" => "%{[json][metadata][namespace]}"
              "[kubernetes][node][name]" => "%{[json][metadata][node_name]}"
              "[metadata][container_name]" => "%{[json][metadata][container_name]}"
              "[metadata][node_name]" => "%{[json][metadata][node_name]}"
              "[metadata][namespace]" => "%{[json][metadata][namespace]}"
              "[metadata][pod_name]" => "%{[json][metadata][pod_name]}"
              "[metadata][pod_uid]" => "%{[json][metadata][pod_uid]}"
              "message" => "%{[json][message]}"
            }
            remove_field => ["type", "host", "json"]
          }
      }

```

Below is the error in case of [type] = "log":

```auto
{
        "_index" : "adp-app-logs-2022.07.20",
        "_type" : "_doc",
        "_id" : "JpSlHIIBBZFaCg47krlI",
        "_score" : 0.0023733466,
        "_source" : {
          "version" : "%{[json][version]}",
          "kubernetes" : {
            "namespace" : "%{[json][metadata][namespace]}",
            "pod" : {
              "uid" : "%{[json][metadata][pod_uid]}",
              "name" : "%{[json][metadata][pod_name]}"
            },
            "node" : {
              "name" : "%{[json][metadata][node_name]}"
            }
          },
          "logplane" : "adp-app-logs",
          "checker" : "value",
          "@timestamp" : "2022-07-20T17:26:09.911Z",
          "filename" : "/logs/logtransformer.log",
          "severity" : "%{[json][severity]}",
          "@version" : "1",
          "metadata" : {
            "namespace" : "%{[json][metadata][namespace]}",
            "container_name" : "%{[json][metadata][container_name]}",
            "pod_name" : "%{[json][metadata][pod_name]}",
            "pod_uid" : "%{[json][metadata][pod_uid]}",
            "node_name" : "%{[json][metadata][node_name]}"
          },
          "service_id" : "%{[json][service_id]}",
          "tags" : [
            "_jsonparsefailure"
          ],
          "message" : [
            "{\"version\": \"1.1.0\", \"timestamp\": \"2022-07-20T17:26:04.694Z\", \"severity\": \"warning\", \"service_id\": \"eric-log-transformer\", \"metadata\" : {\"namespace\": \"zyadros\", \"pod_name\": \"eric-log-transformer-59577c5f7b-24btq\", \"node_name\": \"node-10-63-142-143\", \"pod_uid\": \"58b1e1ef-318a-4a06-ba00-ba0345972b13\", \"container_name\": \"logtransformer\"}, \"message\": \"Error parsing json {:source=>'message', :raw=>' at [Source: (byte[])'{'version': '1.1.0', 'timestamp': '2022-07-20T17:25:50.100Z', 'severity': 'warning', 'service_id': 'eric-log-transformer', 'metadata' : {'namespace': 'zyadros', 'pod_name': 'eric-log-transformer-59577c5f7b-24btq', 'node_name': 'node-10-63-142-143', 'pod_uid': '58b1e1ef-318a-4a06-ba00-ba0345972b13', 'container_name': 'logtransformer'}, 'message': 'Error parsing json {:source=>'message', :raw=>'{\\\\'version\\\\': \\\\'1.1.0\\\\', \\\\'timestamp\\\\': \\\\'2022-07-20T17:25:48.011Z\\\\', \\\\'severity\\\\': \\\\'warning\\\\', \\\\'servi'[truncated 1147 bytes]; line: 1, column: 400]>}\\'}', :exception=>#<LogStash::Json::ParserError: Unrecognized token 'at': was expecting ('true', 'false' or 'null')",
            "%{[json][message]}"
          ]
        }
      }

```

But when applied only a single filter, each one is giving all the values correctly.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 20, 2022, 5:18pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/2 "2022-07-20T17:18:17Z")

</div>

Which filters? You need to provide more context.

Share your logstash configuration with the filters that are not working and also share some sample messages.

Without more information it is not possible to know what is the issue.

---

<div class="post-metadata">

**Author:** ![djrshn2346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djrshn2346/32/108594_2.png) [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\
**Post date:** [July 20, 2022, 5:28pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/3 "2022-07-20T17:28:55Z")

</div>

Added the configuration and the error.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 20, 2022, 7:01pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/4 "2022-07-20T19:01:20Z")

</div>

The `mutate` filter has no order guarantee with `add_field` and `remove_field` , so you should not use it with the same field in the same mutate block.

There is an note in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-proc_order) about it.

> Each mutation must be in its own code block if the sequence of operations needs to be preserved.

Since you need to run `remove_field` after `add_field`, you need to use them in differente mutate blocks.

Remove the field `json` from your `remove_field` and add another mutate block with it after.

```auto
mutate {
    remove_field => ["json"]
}

```

See if this solves your issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 20, 2022, 7:34pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/5 "2022-07-20T19:34:16Z")

</div>

> [@leandrojmp](#):
>
> The `mutate` filter has no order guarantee with `add_field` and `remove_field`

Although it is not documented, the order of the common filter options is fixed. The [code](https://github.com/elastic/logstash/blob/cfbded232c151fbef0f4cda15dc0b8ed1b9c0fbb/logstash-core/lib/logstash/filters/base.rb#L197) processes them in the order add\_field, remove\_field, add\_tag, remove\_tag.

---

<div class="post-metadata">

**Author:** ![djrshn2346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djrshn2346/32/108594_2.png) [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\
**Post date:** [July 21, 2022, 11:23am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/6 "2022-07-21T11:23:43Z")

</div>

Nothing happened, getting another error after adding this:

```auto
mutate { gsub => ["message", "(\W)at(\W)", '\1""\2'] }

```

The error:

```auto
"message" : "{\"version\": \"1.1.0\", \"timestamp\": \"2022-07-21T11:19:08.308Z\", \"severity\": \"warning\", \"service_id\": \"eric-log-transformer\", \"metadata\" : {\"namespace\": \"zyadros\", \"pod_name\": \"eric-log-transformer-846fc647f5-hhj46\", \"node_name\": \"node-10-63-142-142\", \"pod_uid\": \"b9aced00-0645-439c-a548-41ba593b9a97\", \"container_name\": \"logtransformer\"}, \"message\": \"Error parsing json {:source=>'message', :raw=>' \\'\\' [Source: (byte[])'{'version': '1.1.0', 'timestamp': '2022-07-21T11:19:05.628Z', 'severity': 'warning', 'service_id': 'eric-log-transformer', 'metadata' : {'namespace': 'zyadros', 'pod_name': 'eric-log-transformer-846fc647f5-hhj46', 'node_name': 'node-10-63-142-142', 'pod_uid': 'b9aced00-0645-439c-a548-41ba593b9a97', 'container_name': 'logtransformer'}, 'message': 'Error parsing json {:source=>'message', :raw=>' \\\\'\\\\' [Source: (byte[])'{'version': '1.1.0', 'timestamp': '2022-07-21T11:19:03.507Z', 'severity': 'warni'[truncated 588 bytes]; line: 1, column: 400]>}\\'}', :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Source': was expecting ('true', 'false' or 'null')",

```

Even after this getting another `Unrecognized token 'Source'` error and another after another.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 21, 2022, 12:06pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/7 "2022-07-21T12:06:55Z")

</div>

Please share your full configuration and some sample messages to make it possible to try to replicate the issue.

---

<div class="post-metadata">

**Author:** ![djrshn2346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djrshn2346/32/108594_2.png) [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\
**Post date:** [July 22, 2022, 5:15am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/8 "2022-07-22T05:15:23Z")

</div>

I resolved the issue. I moved the `json {}` decode part from the filtering part to the input part, and used `codec => json_lines`. Now this is working perfectly fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2022, 5:16am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-when-all-the-filters-are-applied-at-once-but-working-when-applied-only-one-filter/310186/9 "2022-08-19T05:16:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.