Remove unnecessary documents Logstash

Elastic Stack Logstash
1

Hello everyOne,

I work on several log files that I process with logstash. I divide them into several documents (multiline) and then I extract the information I want.

The problem is that I find myself in the end with several documents where I have nothing interesting and that takes me up space.

Do you know a way to delete documents where there is no information extract by logstash ?

Thank you very much for your help !

2

In logstash, you can use drop.
For instance:

if "toDrop" in [tags]{
  drop {}
}

https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html

3

Hello @Nico-DF,

Thank you for your reply.
I used the drop filter as you told me with the different fields that I extracted logs, but it did the opposite of what I wanted. It only returns empty documents (Logstash deleted me all the documents where there was information).

That is one example of my drop filter :

if "Pilote" in [message]{
drop {} }

4

What is your condition (not that the one you tested, but the one you want) to drop the messages?

5

If no fields are extracted from the document, delete the document.

6

At first I thought it was going in the opposite direction. So: "If we find this field in message, keep it".

7

Two choices then:

  • Either you have, depending of different format (grok filter), always at least one field in common, let's say "id", you can use:
if ![id] {
  drop{}
}

so if the field does not exists, drop the message.

  • Or, in your grok filter, add a tag, let say: "Parsed", and use
if "Parsed" not in [tags] {
  drop{}
}

The condition you used before doesn't really make sense (I think). You can only use "String" in [tags] with tags. For other fields, you must have an equality (or [field] in "String 1, String 2" (check maybe if this is correct, I don't remember well))

8

Okey Thank you for your help.
I have try the second solution. I add one tag in my grok filter like this :

grok {
             break_on_match => false
	     match => {"message" => 'zFlow\(LOCAL\) <- STRING: "(?<Flow>[^"]*)'}
	     match => {"message" => 'Pilote\(LOCAL\) <- STRING: "(?<Pilote>[^"]*)'}
             add_tag => ["Parsed"]
		}

And then I have my drop filter :

if "Parsed" not in [message] {
		drop {}
                            }

But it doesn't work.
I still have a lot of document where my pilot and flow fields are not present.

9

It's: "Parsed" not in [tags]

1 Like
10

Thank you very much @Nico-DF It works !

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.