Log Filter for different cluster

[Naruhodou (Reggie Yang)]

Hi,

I'm collecting logs using Filebeat and Logstash from multiple Kafka clusters. I want to filter this logs based on log-level in Logstash, so my current configuration is like:

  if [log_level] == "TRACE" or [log_level] == "DEBUG"{
    drop {
    }
  }

This will drop TRACE and DEBUG, so it makes the log level INFO.

First I want to know if there are any other ways to filter logs according to log levels.

Now I want different log filter level in different clusters. I've send the clustername as one field in Filebeat, so the Logstash configuration is like this:

    if ([log_level] == "TRACE") and ([clustername] == "Cluster1"){
      drop {
      }
    }

    if ([log_level] == "TRACE" or [log_level] == "DEBUG" or [log_level] == "INFO")  and ([clustername] == "Cluster2"){
      drop {
      }
    }

    else {
      if [log_level] == "TRACE" or [log_level] == "DEBUG"{
        drop {
        }
      }
    }

In this way, cluster 1 and cluster 2 have different log filter levels. As now I'm trying to develop an API to change the log level settings, I found it very difficult to write a script to do this. It needs to find the corresponding line of the cluster and change it. And when I want to add a new cluster, I needs to add another if.

Any suggestions on how to make things simpler?

Thanks!

[Badger]

Do a lookup using a translate filter on the clustername to a tag that tells you the lowest level of logging to retain. Then act based on the tag. You would want a file-based dictionary, not an inline one. Then adding a cluster just means adding one line to the dictionary.

    translate {
        field => "clustername"
        destination => "[@metadata][loglevel2keep]"
        dictionary => {
            "cluster4" => "keepwarn"
            "cluster1" => "keepinfo"
        }
    }
    if [@metadata][loglevel2keep] == "keepwarn" and [log_level] in [ "TRACE", "DEBUG", "INFO" ] { drop {} }
    if [@metadata][loglevel2keep] == "keepinfo" and [log_level] in [ "TRACE", "DEBUG" ] { drop {} }

[Naruhodou (Reggie Yang)]

Great! That's the way I finally came up with. I used a file to store the dict. Thanks Badger!