# Remove unnecessary documents Logstash **URL:** \ **Category:** Logstash\ **Created:** [May 15, 2017, 1:09pm UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768 "2017-05-15T13:09:24Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 15, 2017, 1:09pm UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/1 "2017-05-15T13:09:24Z")
Hello everyOne, I work on several log files that I process with logstash. I divide them into several documents (multiline) and then I extract the information I want. The problem is that I find myself in the end with several documents where I have nothing interesting and that takes me up space. Do you know a way to delete documents where there is no information extract by logstash ? Thank you very much for your help ! ---
**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\ **Post date:** [May 15, 2017, 1:15pm UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/2 "2017-05-15T13:15:10Z")
In logstash, you can use drop. For instance: ```auto if "toDrop" in [tags]{ drop {} } ``` [https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html) ---
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 17, 2017, 7:50am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/3 "2017-05-17T07:50:38Z")
Hello @Nico-DF, Thank you for your reply. I used the drop filter as you told me with the different fields that I extracted logs, but it did the opposite of what I wanted. It only returns empty documents (Logstash deleted me all the documents where there was information). That is one example of my drop filter : > if "Pilote" in [message]{ > drop {} } ---
**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\ **Post date:** [May 17, 2017, 7:53am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/4 "2017-05-17T07:53:16Z")
What is your condition (not that the one you tested, but the one you want) to drop the messages? ---
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 17, 2017, 8:02am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/5 "2017-05-17T08:02:03Z")
If no fields are extracted from the document, delete the document. ---
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 17, 2017, 8:09am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/6 "2017-05-17T08:09:13Z")
At first I thought it was going in the opposite direction. So: "If we find this field in message, keep it". ---
**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\ **Post date:** [May 17, 2017, 8:10am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/7 "2017-05-17T08:10:09Z")
Two choices then: - Either you have, depending of different format (grok filter), always at least one field in common, let's say "id", you can use: ```auto if ![id] { drop{} } ``` so if the field does not exists, drop the message. - Or, in your grok filter, add a tag, let say: "Parsed", and use ```auto if "Parsed" not in [tags] { drop{} } ``` The condition you used before doesn't really make sense (I think). You can only use `"String" in [tags]` with tags. For other fields, you must have an equality (or `[field] in "String 1, String 2"` (check maybe if this is correct, I don't remember well)) ---
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 17, 2017, 9:02am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/8 "2017-05-17T09:02:17Z")
Okey Thank you for your help. I have try the second solution. I add one tag in my grok filter like this : ``` grok { break_on_match => false match => {"message" => 'zFlow\(LOCAL\) <- STRING: "(?[^"]*)'} match => {"message" => 'Pilote\(LOCAL\) <- STRING: "(?[^"]*)'} add_tag => ["Parsed"] } ``` And then I have my drop filter : ``` if "Parsed" not in [message] { drop {} } ``` But it doesn't work. I still have a lot of document where my pilot and flow fields are not present. ---
**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\ **Post date:** [May 17, 2017, 9:04am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/9 "2017-05-17T09:04:04Z")
> [@Sam67000](#): > > "Parsed" not in [message] It's: `"Parsed" not in [tags]` ---
**Author:** ![Sam67000](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Post date:** [May 17, 2017, 11:54am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/10 "2017-05-17T11:54:09Z")
Thank you very much @Nico-DF It works ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 14, 2017, 11:54am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768/11 "2017-06-14T11:54:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.