# Logstash filtering issue

**URL:** <https://discuss.elastic.co/t/logstash-filtering-issue/96436>\
**Category:** Logstash\
**Created:** [August 9, 2017, 1:01pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436 "2017-08-09T13:01:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![himaz.m](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@himaz.m](https://discuss.elastic.co/u/himaz.m)\
**Post date:** [August 9, 2017, 1:01pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436/1 "2017-08-09T13:01:15Z")

</div>

I'm using filebeat, logstash and kibana.

I see an issue in Kibana which looks like the logstash grok pattern was not applied for some of the log lines. I don't see an issue in the log lines even, both filtered and unfiltered log lines looks similar. I've added a sample below.

My log file looks like this (1st line was not filtered and second line was filtered properly)

```
2017-08-08 23:57:40.625+0000 | INFO | CONVEN_XXXXXX | PROD | ae92e5992cf7a5e7 | bca227c9d4848566 | 9778 | [http-nio-5000-exec-6] | c.v.c.a.product.ProductServiceImpl | Returning Product Reviews Response
2017-08-08 23:57:40.625+0000 | INFO | CONVEN_XXXXXX | PROD | ae92e5992cf7a5e7 | bca227c9d4848566 | 9778 | [http-nio-5000-exec-6] | c.v.c.infrastructure.RestClient | GET XXXXProductReviews Response : {"data":[],"totalCount":0,"dateStamp":"2017-08-08T16:57:40+0000","errorCodes":[],"success":true,"vmid":"331024"}

```

My Logstash configuration is

```
input {
 beats {
  port => 5044
  codec => multiline {
   pattern => "(^%{TIMESTAMP_ISO8601})"
   negate => true
   what => "previous"
  }
 }
}

filter {
    if [type] == "mixlog" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \| %{LOGLEVEL:loglevel} \| %{DATA:module} \| %{DATA:environment} \| %{DATA:traceid} \| %{DATA:spanid} \| %{DATA:processid} \| \[%{DATA:thread}\] \| %{DATA:class} \| %{GREEDYDATA:message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
      overwrite => ["message"]
    }
    date {
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSSZ", "ISO8601"]
    }
  }
  
  
}

output {
    amazon_es {
        hosts => ["xxxxxxxx.xxxxxx.xxxxxxxx.com"]
        region => "us-east-2"
    }
}

# test elasticsearch:
# curl -X GET 'https://xxxxxxxx.xxxxxx.xxxxxxxx.com/logstash-2017.05.25/_search?pretty&q=response:200'

```

This is how the 1st line looks in Kibana. I dont even see the beat properties (beat.hostname, beat.version,etc,).

 ![Kibana-1](https://us1.discourse-cdn.com/elastic/original/3X/6/9/6991850e038e14456b45c7dbb2edfd7f99556687.png)

This is how the 2nd line looks in Kibana. This is the expected behavior.

 ![Kibana-2](https://us1.discourse-cdn.com/elastic/original/3X/3/4/34b19cf71b6c6eea9ce4798026efd4cb106c6e3e.png)

Are there any configuration issues? Please help me!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 9, 2017, 1:14pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436/2 "2017-08-09T13:14:26Z")

</div>

You only filter "mixlog" events but the first line has the type "logs".

---

<div class="post-metadata">

**Author:** ![himaz.m](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@himaz.m](https://discuss.elastic.co/u/himaz.m)\
**Post date:** [August 9, 2017, 1:23pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436/3 "2017-08-09T13:23:09Z")

</div>

@magnusbaeck Good catch! Thanks.  
But in my filebeat I've set the type as "mixlog". How come it is showing as "logs"?

Also I found another log entry which has the type as "mixlog", but didn't filtered properly. Why is that?

 ![Kibana-3](https://us1.discourse-cdn.com/elastic/original/3X/6/1/618e966a80d88146e42c22b21040d479b8ec854b.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 9, 2017, 6:43pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436/4 "2017-08-09T18:43:45Z")

</div>

> But in my filebeat I’ve set the type as “mixlog”.

How? Show your configuration. **Format it as preformatted text so the YAML markup doesn't get mangled.**

> Also I found another log entry which has the type as “mixlog”, but didn’t filtered properly. Why is that?

I don't know.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 6, 2017, 6:44pm UTC](https://discuss.elastic.co/t/logstash-filtering-issue/96436/5 "2017-09-06T18:44:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.