Use logcheck ignore rules

Elastic Stack Logstash
#1

I've started using ELK yesterday, moving from logcheck and I would like to re-use the ignore rules of logcheck and some custom I had added. I currently have two servers providing logs throught filebeat to logstash, who then passes them on to elasticsearch and kibana.

I see two ways of doing this, either by filtering the logs before giving them to elasticsearch (or kibana?*) or by somehow enable a data view of some sort which doesn't include the ignored lines on searches and so on.

I've seen that logstash has grep filters, could you point me to a good tutorial on how to use it? And maybe give your 5 cents on the proper way to do this.

What would you recommend?

*I am a newbie here, I am still illiterate about who is supposed to do what, not to say the proper terminology.

#2

The grep filter was deprecated years ago. This is how you'd typically drop boring events:

filter {
  if [message] =~ /regexp matching event I don't care about/ {
    drop { }
  }
}
#3

Excellent!

What about reading the regexp from a file or files? I could generate such
filter with a script but that would make a huge 'if' containing all
logcheck's drop rules.

Cheers,

Pol

#4

You might be able to use the translate filter, but otherwise generating a configuration file is probably your best bet. Keep in mind that this can significantly affect the event throughput.

#5

Thanks Magnus for the prompt reply.

Could you elaborate on what do you mean as for configuration file? Are you
referring to pre-filter syslog somehow before sending it via filebeat or
the drop filter you mentioned on your first post?

I was indeed pointing at the performance impact filtering in that way might
have on my previous comment. I don't know if it would be of concern.

Cheers,

Pol

ps.
As a side quest, I think I am more inclined now to publish all logs but
have a default view on the stored data with the boring messages filtered
out, but being able to switch the filter off to see a suspicious log
message in context.

#6

Could you elaborate on what do you mean as for configuration file? Are you
referring to pre-filter syslog somehow before sending it via filebeat or
the drop filter you mentioned on your first post?

Whichever works best, really.

#7