Remove unnecessary documents Logstash

Hello everyOne,

I work on several log files that I process with logstash. I divide them into several documents (multiline) and then I extract the information I want.

The problem is that I find myself in the end with several documents where I have nothing interesting and that takes me up space.

Do you know a way to delete documents where there is no information extract by logstash ?

Thank you very much for your help !

In logstash, you can use drop.
For instance:

if "toDrop" in [tags]{
  drop {}

Hello @Nico-DF,

Thank you for your reply.
I used the drop filter as you told me with the different fields that I extracted logs, but it did the opposite of what I wanted. It only returns empty documents (Logstash deleted me all the documents where there was information).

That is one example of my drop filter :

if "Pilote" in [message]{
drop {} }

What is your condition (not that the one you tested, but the one you want) to drop the messages?

If no fields are extracted from the document, delete the document.

At first I thought it was going in the opposite direction. So: "If we find this field in message, keep it".

Two choices then:

  • Either you have, depending of different format (grok filter), always at least one field in common, let's say "id", you can use:
if ![id] {

so if the field does not exists, drop the message.

  • Or, in your grok filter, add a tag, let say: "Parsed", and use
if "Parsed" not in [tags] {

The condition you used before doesn't really make sense (I think). You can only use "String" in [tags] with tags. For other fields, you must have an equality (or [field] in "String 1, String 2" (check maybe if this is correct, I don't remember well))

Okey Thank you for your help.
I have try the second solution. I add one tag in my grok filter like this :

grok {
             break_on_match => false
	     match => {"message" => 'zFlow\(LOCAL\) <- STRING: "(?<Flow>[^"]*)'}
	     match => {"message" => 'Pilote\(LOCAL\) <- STRING: "(?<Pilote>[^"]*)'}
             add_tag => ["Parsed"]

And then I have my drop filter :

if "Parsed" not in [message] {
		drop {}

But it doesn't work.
I still have a lot of document where my pilot and flow fields are not present.

It's: "Parsed" not in [tags]

1 Like

Thank you very much @Nico-DF It works !

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.