# Grok ingest pipeline

**URL:** <https://discuss.elastic.co/t/grok-ingest-pipeline/317708>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [October 28, 2022, 8:02pm UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708 "2022-10-28T20:02:21Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Walter\_Hiranpat1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walter_hiranpat1/32/112337_2.png) [@Walter\_Hiranpat1](https://discuss.elastic.co/u/Walter_Hiranpat1)\
**Post date:** [October 28, 2022, 8:02pm UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/1 "2022-10-28T20:02:21Z")

</div>

I was wondering if there a way to accept all patterns excluding a specific regular expression?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [October 29, 2022, 4:55am UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/2 "2022-10-29T04:55:16Z")

</div>

Can u be more specific? Can u provide an example of what you're trying to do?

---

<div class="post-metadata">

**Author:** ![Walter\_Hiranpat1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walter_hiranpat1/32/112337_2.png) [@Walter\_Hiranpat1](https://discuss.elastic.co/u/Walter_Hiranpat1)\
**Post date:** [October 31, 2022, 4:33pm UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/3 "2022-10-31T16:33:19Z")

</div>

I can't share an example but I try to be more specific. I have setup a regex in the ingest-pipeline using the grok to look for a specific message. But I was misinformed, they want all data excluding that message format. So I am trying to see if there is a way to just accept everything that doesn't match that message and exclude the ones that do match the original pattern. I am wondering if it should be a conditional statement or maybe I can encapsulate the pattern with '!'?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [October 31, 2022, 5:17pm UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/4 "2022-10-31T17:17:33Z")

</div>

I think i get it but are you saying that you want to drop any documents that don't match the GROK pattern (Or do)? or do something else based on the pattern?

---

<div class="post-metadata">

**Author:** ![Walter\_Hiranpat1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walter_hiranpat1/32/112337_2.png) [@Walter\_Hiranpat1](https://discuss.elastic.co/u/Walter_Hiranpat1)\
**Post date:** [October 31, 2022, 5:59pm UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/5 "2022-10-31T17:59:18Z")

</div>

Yes, The grok pattern that I have shouldn't be indexed. They kinda don't serve any purpose in our configuration. I think if they are drop that would be more ideal since it would reduce stress to our storage.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 1, 2022, 2:07am UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/6 "2022-11-01T02:07:25Z")

</div>

If you're using filebeat, you can use a drop processor with a regex.

You can also use an ingest pipeline with a drop processor and a condition using RegEx for that as well. (Be careful a bit with that)

That can be a little tricky.

> **[Ingest pipelines | Elasticsearch Guide \[8.4\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-run-processor)**

And yes you could set a grok And then set a tag in that grok processor and then do the drop processor afterwards based on that tag.

So you have options? You just got to figure out what's the most efficient for your use case

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2022, 2:08am UTC](https://discuss.elastic.co/t/grok-ingest-pipeline/317708/7 "2022-11-29T02:08:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
