Trouble filtering out logs

Elastic Stack Logstash
1

I'm trying to filter out certain types of logs which contain a string message. The error is a log level error which consumes my all events view in kibana. I would like to not have this appear in kibana. I would also be open to changing the level to debug in place of dropping.

I've tried the following with no success:

Adding a filter to my conf.d file on my log server:

  if "part of my string" in [msg] {
    drop {}
  }
  if "part of other string" in [msg] {
    drop {}
  }

I've also tried adding an exclude_lines filter under my filebeat.prospectors file:
exclude_lines: ['.partofmystring.', '.partofotherstring::.']

And I've tired using a processor:
filebeat.prospectors:
-input_type: log
paths:
/my/path/log
processors:
-drop_event:
when:
contains:
json: 'part of my string'
json._source: 'part of other string'
_source.msg: 'part of other string'
.msg: 'part of other string'

None of these have worked as anticipated. How should I go about filtering these non-important logs from my kibana dashboard?

2

Try if this works for you:

if [msg] =~ "part of my string" {
  drop { }
}
3

Please show an example of an unwanted event that you wanted to drop. Copy/paste from Kibana's JSON tab.

4

{
"_index": "event-2018.03.07",
"_type": "json-events",
"_id": "AWIBbXhybvXlGNlImwCZ",
"_version": 1,
"_score": null,
"_source": {
"msg": "• using configured layout:: layout (located @ "/var/app/current/views/layout")",
"cluster": "production",
"offset": 3287370,
"level": "20",
"input_type": "log",
"beats_input": "secure",
"pid": "4103",
"source": "/var/app/current/log/irisaft.log",
"message": "{"name":"irisaft","hostname":"ip-172-30-0-134","pid":4103,"level":20,"msg":"• using configured layout:: layout (located @ \"/var/app/current/views/layout\")","time":"2018-03-07T17:06:12.796Z","v":0}",
"type": "json-events",
"tags": [
"beats_input_codec_plain_applied"
],
"hostname": "",
"@timestamp": "2018-03-07T17:06:12.796Z",
"@version": "1",
"beat": {
"name": "",
"hostname": "",
"version": "5.6.5"
},
"host": ""
"name": "irisaft"
},
"fields": {
"@timestamp": [
1520442372796
]
},
"highlight": {
"cluster": [
"@kibana-highlighted-field@production@/kibana-highlighted-field@"
]
},
"sort": [
1520442372796
]
}

5

Okay, looks promising. What does your full configuration look like?

6 
   filter {
      if [type] in ["json", "json-events", "json-access"] {
      json {
          source => "message"
          target => "parsed"
          add_field => {
              "level" => "%{parsed[level]}"
              "name" => "%{parsed[name]}"
              "hostname" => "%{parsed[hostname]}"
              "pid" => "%{parsed[pid]}"
              "msg" => "%{parsed[msg]}"
              "time" => "%{parsed[time]}"
          }
      }
      if [msg] =~ "Rendering view" {
        drop {}
      }
      if [msg] =~ "using configured layout" {
        drop {}
      }
      }

  if [type] == "json-access" and "_jsonparsefailure" not in [tags] {
  mutate {
      add_field => {
          "method" => "%{parsed[req][method]}"
          "url" => "%{parsed[req][url]}"
          "status_code" => "%{parsed[res][statusCode]}"
      }
  }

  if [parsed][responseTime] {
      mutate {
          add_field => {
              "response_time" => "%{parsed[responseTime]}"
          }
      }
  }
  }

  if [type] in ["json", "json-events", "json-access"] and "_jsonparsefailure" not in [tags] {
  if [msg] =~ "Rendering view" {
    drop {}
  }
  if [msg] =~ "using configured layout" {
    drop {}
  }
  date {
      match => ["time", "ISO8601"]
  }

  mutate {
      remove_field => ["parsed", "time"]
  }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => false
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
7

Okay, I'm not sure what's up here. I'd verify that Logstash really is running with the configuration you think it is and simplify the configuration to a minimal example.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.