# Log Filter for different cluster

**URL:** <https://discuss.elastic.co/t/log-filter-for-different-cluster/144766>\
**Category:** Logstash\
**Created:** [August 16, 2018, 6:39pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766 "2018-08-16T18:39:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [August 16, 2018, 6:39pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766/1 "2018-08-16T18:39:37Z")

</div>

Hi,

I'm collecting logs using Filebeat and Logstash from multiple Kafka clusters. I want to filter this logs based on log-level in Logstash, so my current configuration is like:

```auto
  if [log_level] == "TRACE" or [log_level] == "DEBUG"{
    drop {
    }
  }

```

This will drop TRACE and DEBUG, so it makes the log level INFO.

First I want to know if there are any other ways to filter logs according to log levels.

Now I want different log filter level in different clusters. I've send the clustername as one field in Filebeat, so the Logstash configuration is like this:

```auto
    if ([log_level] == "TRACE") and ([clustername] == "Cluster1"){
      drop {
      }
    }

    if ([log_level] == "TRACE" or [log_level] == "DEBUG" or [log_level] == "INFO") and ([clustername] == "Cluster2"){
      drop {
      }
    }

    else {
      if [log_level] == "TRACE" or [log_level] == "DEBUG"{
        drop {
        }
      }
    }

```

In this way, cluster 1 and cluster 2 have different log filter levels. As now I'm trying to develop an API to change the log level settings, I found it very difficult to write a script to do this. It needs to find the corresponding line of the cluster and change it. And when I want to add a new cluster, I needs to add another if.

Any suggestions on how to make things simpler?

Thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 16, 2018, 6:51pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766/2 "2018-08-16T18:51:25Z")

</div>

Do a lookup using a translate filter on the clustername to a tag that tells you the lowest level of logging to retain. Then act based on the tag. You would want a file-based dictionary, not an inline one. Then adding a cluster just means adding one line to the dictionary.

```
    translate {
        field => "clustername"
        destination => "[@metadata][loglevel2keep]"
        dictionary => {
            "cluster4" => "keepwarn"
            "cluster1" => "keepinfo"
        }
    }
    if [@metadata][loglevel2keep] == "keepwarn" and [log_level] in ["TRACE", "DEBUG", "INFO"] { drop {} }
    if [@metadata][loglevel2keep] == "keepinfo" and [log_level] in ["TRACE", "DEBUG"] { drop {} }
```

---

<div class="post-metadata">

**Author:** ![Naruhodou](https://avatars.discourse-cdn.com/v4/letter/n/7feea3/32.png) [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\
**Post date:** [August 21, 2018, 11:06pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766/3 "2018-08-21T23:06:30Z")

</div>

Great! That's the way I finally came up with. I used a file to store the dict. Thanks Badger!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2018, 11:06pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766/4 "2018-09-18T23:06:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.