# Grok filter output to elastic not filtered

**URL:** <https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898>\
**Category:** Logstash\
**Created:** [September 18, 2017, 4:19pm UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898 "2017-09-18T16:19:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fridolf](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fridolf](https://discuss.elastic.co/u/fridolf)\
**Post date:** [September 18, 2017, 4:19pm UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898/1 "2017-09-18T16:19:41Z")

</div>

Using logstash 5.4.0 and elastic 5.4.0

I have a grok filter:  
%{MYDATESTAMP:loggedtimestamp} GMT%{ISO8601\_TIMEZONE} %{MYTZ} [-] %{LOGLEVEL:loglevel}: %{GREEDYDATA:logmessage}

I have below patterns:  
MYTZ (CE[S]?T)  
MYDATESTAMP %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME}

My config file looks like below:

input {  
file {  
path =\> "/root/.pm2/logs/mylog-test.log"  
start\_position =\> "beginning"  
}  
}  
filter {  
grok {  
patterns\_dir =\> ["opt/logstash/patterns"]  
match =\> { "message" =\> "%{MYDATESTAMP:loggedtimestamp} GMT%{ISO8601\_TIMEZONE} %{MYTZ} [-] %{LOGLEVEL:loglevel}: %{GREEDYDATA:logmessage}" }

```
}

```

}

# Output to elastic

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> testlogs  
}  
}

Logfile entries look like below:  
2017-09-07T09:07:07.773Z POST /api/Account/Login 200  
Thu Sep 14 2017 05:59:55 GMT+0000 (CEST) - info: User [testuser@gmail.com](mailto:testuser@gmail.com) logged in

When testing this online it works, and only picks the lines having format like the second entry.  
[http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result)

But when running this with logstash elastic search gets filled with all entries. Some of the ones that should not be there have tags grokparsefailure, some dont. The once not matching the filter do not have the correct fields like loggedtimestamp.

Any suggestions to what is wrong with my config file/grok filter?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2017, 5:26am UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898/2 "2017-09-20T05:26:33Z")

</div>

Unless you explicitly configure Logstash otherwise, all events will be sent to all outputs. If you don't want events where the grok parsing failed to be sent to your elasticsearch output, wrap it in a conditional.

```nohighlight
if "_grokparsefailure` not in [tags] {
  elasticsearch { ... }
}

```

---

<div class="post-metadata">

**Author:** ![fridolf](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fridolf](https://discuss.elastic.co/u/fridolf)\
**Post date:** [September 20, 2017, 8:02am UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898/3 "2017-09-20T08:02:14Z")

</div>

After writing this post I relized that must be the case, but I could not really find that information anywhere in the logstash documentation. That would have been nice. I used an If statement and dropped the entries I was not interested in. Like this  
if [message] !~ "logged in" {  
drop { }  
}  
That seems to work fine.  
Thank you for replying!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 18, 2017, 8:03am UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898/4 "2017-10-18T08:03:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.