How to exclude bad output (lines not matching 'grok' pattern) from logstash?

Elastic Stack Logstash
1

I have a log file and I am parsing it through Logstash and storing it in some place. But the the problem is that some lines in the log file do not always match my grok pattern and are therefore tagged as 'grokparsefailure' etc automatically. I do not tag any line explicitly, so the lines which are automatically tagged by logstash are the wrongly structured lines and I want these lines to be skipped i.e. I don't want these output lines to appear in the output at all (lines which have tags).

Can anyone help me to achieve this?
Please help!

2

Use the drop filter to, well, drop events you don't want.

if "_grokparsefailure" in [tags] {
  drop { }
}
6 Likes
3

Thanks a lot @magnusbaeck! It totally worked!:slightly_smiling:

4

hey guys, so i add it on block filter and after grok or where, pls reply soon. thanks you so much

5

@Tr_ng_Trang, please open a new thread and supply more details.

6

This construct is present in many examples online, but what if I still want the line logged, just differently?

If my grok-filter failed, I do not want any other filters applied, but still want to record the entire message in a separate output -- how would I achieve that?

The separate output part is easy -- by using checking for the [tags] -- but what about avoiding all the other filters without dropping the event? Must I move them all into a condition checking for _grokparsefailure not being among the [tags], or is there some other way? Thank you!

7

Must I move them all into a condition checking for _grokparsefailure not being among the [tags]

Yes, that's the way to do it.

8

@magnusbaeck this method works for me too, but I want to know why the line with _grokparsefailure line are still sent to ES. I am new to use logstash and ES, from my perspective, the filter is to filter some lines which will be grok-failed and not sent to ES, is there something wrong, thanks a lot.

9

Please start a new thread and provide additional details about your configuration.