# Trouble filtering out logs

**URL:** <https://discuss.elastic.co/t/trouble-filtering-out-logs/122792>\
**Category:** Logstash\
**Created:** [March 6, 2018, 10:03pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792 "2018-03-06T22:03:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aleita](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aleita/32/28508_2.png) [@Aleita](https://discuss.elastic.co/u/Aleita)\
**Post date:** [March 6, 2018, 10:03pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/1 "2018-03-06T22:03:48Z")

</div>

I'm trying to filter out certain types of logs which contain a string message. The error is a log level error which consumes my all events view in kibana. I would like to not have this appear in kibana. I would also be open to changing the level to debug in place of dropping.

I've tried the following with no success:

**Adding a filter to my conf.d file on my log server:**

```
  if "part of my string" in [msg] {
    drop {}
  }
  if "part of other string" in [msg] {
    drop {}
  }

```

**I've also tried adding an exclude\_lines filter under my filebeat.prospectors file:**  
exclude\_lines: ['._partofmystring._', '._partofotherstring::._']

**And I've tired using a processor:**  
filebeat.prospectors:  
-input\_type: log  
paths:  
/my/path/log  
processors:  
-drop\_event:  
when:  
contains:  
json: 'part of my string'  
json.\_source: 'part of other string'  
\_source.msg: 'part of other string'  
.msg: 'part of other string'

None of these have worked as anticipated. How should I go about filtering these non-important logs from my kibana dashboard?

---

<div class="post-metadata">

**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\
**Post date:** [March 6, 2018, 11:13pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/2 "2018-03-06T23:13:38Z")

</div>

Try if this works for you:

```auto
if [msg] =~ "part of my string" {
  drop { }
}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2018, 1:14pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/3 "2018-03-07T13:14:01Z")

</div>

Please show an example of an unwanted event that you wanted to drop. Copy/paste from Kibana's JSON tab.

---

<div class="post-metadata">

**Author:** ![Aleita](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aleita/32/28508_2.png) [@Aleita](https://discuss.elastic.co/u/Aleita)\
**Post date:** [March 7, 2018, 5:08pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/4 "2018-03-07T17:08:02Z")

</div>

{  
"\_index": "event-2018.03.07",  
"\_type": "json-events",  
"\_id": "AWIBbXhybvXlGNlImwCZ",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"msg": "• using configured layout:: layout (located @ "/var/app/current/views/layout")",  
"cluster": "production",  
"offset": 3287370,  
"level": "20",  
"input\_type": "log",  
"beats\_input": "secure",  
"pid": "4103",  
"source": "/var/app/current/log/irisaft.log",  
"message": "{"name":"irisaft","hostname":"ip-172-30-0-134","pid":4103,"level":20,"msg":"• using configured layout:: layout (located @ \"/var/app/current/views/layout\")","time":"2018-03-07T17:06:12.796Z","v":0}",  
"type": "json-events",  
"tags": [  
"beats\_input\_codec\_plain\_applied"  
],  
"hostname": "",  
"@timestamp": "2018-03-07T17:06:12.796Z",  
"@version": "1",  
"beat": {  
"name": "",  
"hostname": "",  
"version": "5.6.5"  
},  
"host": ""  
"name": "irisaft"  
},  
"fields": {  
"@timestamp": [  
1520442372796  
]  
},  
"highlight": {  
"cluster": [  
"@kibana-highlighted-field@production@/kibana-highlighted-field@"  
]  
},  
"sort": [  
1520442372796  
]  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2018, 8:22pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/5 "2018-03-07T20:22:06Z")

</div>

Okay, looks promising. What does your full configuration look like?

---

<div class="post-metadata">

**Author:** ![Aleita](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aleita/32/28508_2.png) [@Aleita](https://discuss.elastic.co/u/Aleita)\
**Post date:** [March 7, 2018, 8:26pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/6 "2018-03-07T20:26:00Z")

</div>

```
   filter {
      if [type] in ["json", "json-events", "json-access"] {
      json {
          source => "message"
          target => "parsed"
          add_field => {
              "level" => "%{parsed[level]}"
              "name" => "%{parsed[name]}"
              "hostname" => "%{parsed[hostname]}"
              "pid" => "%{parsed[pid]}"
              "msg" => "%{parsed[msg]}"
              "time" => "%{parsed[time]}"
          }
      }
      if [msg] =~ "Rendering view" {
        drop {}
      }
      if [msg] =~ "using configured layout" {
        drop {}
      }
      }

  if [type] == "json-access" and "_jsonparsefailure" not in [tags] {
  mutate {
      add_field => {
          "method" => "%{parsed[req][method]}"
          "url" => "%{parsed[req][url]}"
          "status_code" => "%{parsed[res][statusCode]}"
      }
  }

  if [parsed][responseTime] {
      mutate {
          add_field => {
              "response_time" => "%{parsed[responseTime]}"
          }
      }
  }
  }

  if [type] in ["json", "json-events", "json-access"] and "_jsonparsefailure" not in [tags] {
  if [msg] =~ "Rendering view" {
    drop {}
  }
  if [msg] =~ "using configured layout" {
    drop {}
  }
  date {
      match => ["time", "ISO8601"]
  }

  mutate {
      remove_field => ["parsed", "time"]
  }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => false
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2018, 8:40pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/7 "2018-03-07T20:40:08Z")

</div>

Okay, I'm not sure what's up here. I'd verify that Logstash really is running with the configuration you think it is and simplify the configuration to a minimal example.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2018, 8:40pm UTC](https://discuss.elastic.co/t/trouble-filtering-out-logs/122792/8 "2018-04-04T20:40:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.