How to exclude bad output (lines not matching 'grok' pattern) from logstash?


(Punit Naik) #1

I have a log file and I am parsing it through Logstash and storing it in some place. But the the problem is that some lines in the log file do not always match my grok pattern and are therefore tagged as 'grokparsefailure' etc automatically. I do not tag any line explicitly, so the lines which are automatically tagged by logstash are the wrongly structured lines and I want these lines to be skipped i.e. I don't want these output lines to appear in the output at all (lines which have tags).

Can anyone help me to achieve this?
Please help!


Grok filter for selecting and formatting certain logs lines
Drop logs from particular hostname
(Magnus Bäck) #2

Use the drop filter to, well, drop events you don't want.

if "_grokparsefailure" in [tags] {
  drop { }
}

(Punit Naik) #3

Thanks a lot @magnusbaeck! It totally worked!:slightly_smiling:


(Trương Trang) #4

hey guys, so i add it on block filter and after grok or where, pls reply soon. thanks you so much


(Magnus Bäck) #5

@Tr_ng_Trang, please open a new thread and supply more details.


(United Marsupials) #6

This construct is present in many examples online, but what if I still want the line logged, just differently?

If my grok-filter failed, I do not want any other filters applied, but still want to record the entire message in a separate output -- how would I achieve that?

The separate output part is easy -- by using checking for the [tags] -- but what about avoiding all the other filters without dropping the event? Must I move them all into a condition checking for _grokparsefailure not being among the [tags], or is there some other way? Thank you!


(Magnus Bäck) #7

Must I move them all into a condition checking for _grokparsefailure not being among the [tags]

Yes, that's the way to do it.