# Drop docs that meet certain Grok pattern **URL:** \ **Category:** Elasticsearch\ **Tags:** ingest-pipeline\ **Created:** [March 16, 2023, 10:45pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889 "2023-03-16T22:45:52Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![demonsquatch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demonsquatch/32/88611_2.png) [@demonsquatch](https://discuss.elastic.co/u/demonsquatch)\ **Post date:** [March 16, 2023, 10:45pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/1 "2023-03-16T22:45:52Z")
Hi all, Currently looking to drop any documents that meet a certain Grok pattern, but am not having much luck on finding anything. In this scenario, I would like do drop anything that matches the pattern of EVENT1 in the below example My current Grok patterns are: ```auto { "EVENT1": "%{TIMESTAMP_ISO8601}(( '%{NOTSPACE:event.action}' %{POSINT})|:) %{LOGLEVEL:log.level} %{GREEDYDATA:message}", "EVENT2": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\] \[A ticket for (%{DATA:vm.name}) of""", "EVENT3": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\] \[(%{DATA:vm.name}) on""", "EVENT4": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\]""", "VSPHERE_USER": """((%{HOSTNAME:user.domain}\\)?%{USERNAME:user.name}|)""" } ``` Thanks in advance for any assistance! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 16, 2023, 11:35pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/2 "2023-03-16T23:35:19Z")
Hi @demonsquatch We should probably let @Badger answer but you certainly can just add a tag when it matches and then use a drop filter conditional on that tag. - Separate into two groks - Set the tag in the first one. - Then conditional grok not on that on the second set of patterns - Then drop based on the tag ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 16, 2023, 11:56pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/3 "2023-03-16T23:56:43Z")
As @stephenb says, you could use a tag, but you assuming event.action will not exist unless the grok creates it you could use that ``` grok { match => { "message" => "%{TIMESTAMP_ISO8601}(( '%{NOTSPACE:event.action}' %{POSINT})|:) %{LOGLEVEL:log.level} %{GREEDYDATA:message}" } } if [event.action] { drop {} } grok { match => { "message" => [an array, the rest of the patterns] } } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 17, 2023, 1:02am UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/4 "2023-03-17T01:02:35Z")
Also, are you using Logstash or Elasticsearch ingest pipelines? The post is Logstash forum, but it has the tag `ingest-pipeline`, so it is a little confusing. I'm asking because if you are using Logstash and want to create ECS fields, as it seems to be the goal, you need to reference the fields using `[event][action]`, `[user][name]`, `[log][level]` etc. In your grok you are using `event.action`, `user.name` etc, on Logstash those are not the same thing. For example, using `user.name` in the following pattern `%{USERNAME:user.name}` will create a field with a literal dot in the name. ```auto { "user.name": "username" } ``` The ECS field is a object named `user` with a nested field named `name`: ```auto { "user": { "name": "username" } } ``` To create a field like this in Logstash you need to use `%{USERNAME:[user][name]}` in this example. ---
**Author:** ![demonsquatch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demonsquatch/32/88611_2.png) [@demonsquatch](https://discuss.elastic.co/u/demonsquatch)\ **Post date:** [March 17, 2023, 2:17am UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/5 "2023-03-17T02:17:57Z")
Thank you for the response and pointing that out - I am doing it with Elasticsearch ingest pipeline and have moved it to the correct forum ---
**Author:** ![demonsquatch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demonsquatch/32/88611_2.png) [@demonsquatch](https://discuss.elastic.co/u/demonsquatch)\ **Post date:** [March 17, 2023, 2:32am UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/6 "2023-03-17T02:32:42Z")
Worked beautifully, thank you so much! ---
**Author:** ![jba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jba/32/118482_2.png) [@jba](https://discuss.elastic.co/u/jba)\ **Post date:** [March 18, 2023, 6:41pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/7 "2023-03-18T18:41:25Z")
You are not the only one that is/was confused. A colleague and I spent at least 5 minutes in a discussion with an Elastic sales representative about pipelines before it dawned on us that she was talking about Elasticsearch Ingest Pipelines and we were inquirering about Logstash Pipelines and why and when it would be advantageous to have more than one. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 15, 2023, 6:41pm UTC](https://discuss.elastic.co/t/drop-docs-that-meet-certain-grok-pattern/327889/8 "2023-04-15T18:41:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.