# Grok Trying to continue with pattern based on match

**URL:** <https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185>\
**Category:** Logstash\
**Created:** [June 18, 2019, 6:04am UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185 "2019-06-18T06:04:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![talqadi7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talqadi7/32/43703_2.png) [@talqadi7](https://discuss.elastic.co/u/talqadi7)\
**Post date:** [June 18, 2019, 6:04am UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/1 "2019-06-18T06:04:58Z")

</div>

Hi, I'm still new to grok and logstash, and appreciate any help. So, I'm currently setting up a grok pattern, to extract a field from a large message field with a lot of human text. The data is extremely unstructured, so I am attempting to extract the source number from the source id, which looks like S012345.  
The pattern i setup:  
grok {  
match =\> {  
"message" =\> "source id%{GREEDYDATA:name}[sS]%{NUMBER:source\_id}"  
}  
}  
However, for 50% of the cases the source id is turning out to be the current index's ID, due to the word source id being present sometimes in paragraphs.  
My question comes here: Is there anyway to insert a condition within grok, to continue to search for a match if the source\_id == index\_id? I've been looking at the syntax for a while, but couldn't really find a similar scenario. Thank you for your help.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [June 18, 2019, 8:56am UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/2 "2019-06-18T08:56:09Z")

</div>

Hi Talal,

Some sample data would provide a little clarity.

> [@talqadi7](#):
>
> index\_id

Where is this index\_id field?

---

<div class="post-metadata">

**Author:** ![talqadi7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talqadi7/32/43703_2.png) [@talqadi7](https://discuss.elastic.co/u/talqadi7)\
**Post date:** [June 19, 2019, 7:32pm UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/3 "2019-06-19T19:32:21Z")

</div>

Hi NerdSec, Thank you very much for your time.  
The index\_id field is ingested from the data. While the source\_ID will be extracted from the message field.The JSON template from the data includes 3 fields, Applicant field, the Index\_ID field, and the message field. The message field is extremely unstructured throughout.  
A sample message field where the pattern makes a mistake would be:  
"The specifications statement is identical to the Source\_ID device.  
RE: S123039  
Index\_ID S123039  
Source\_ID Electrical Appliance S120492  
Dated October 6, 2016  
Received October 7, 2016"  
In this case, the source\_id is being extracted as S123039 because it is present after source\_id is mentioned in text. However, I can't be more specific as there are many different formats for what comes between the **Source\_ID** and the **S120492** (sometimes a name, sometimes more information). Although the index\_ID is present in this message field, it is also in the data itself, when ingested. Is there any possibility to compare the values, and if they are equal to skip the match.

I hope this provides some clarity.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [June 20, 2019, 5:15am UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/4 "2019-06-20T05:15:49Z")

</div>

Hi Talal,

If you are receiving messages in json, then you should use the `json` codec in your input.

Or you can also use the `json` filter. [JSON filter plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)

Once you have used this filter, you will now have the unstructured data in the **message** (Or whatever is the key name) field available to you separately. The question is, does the new field contain multiline data as mentioned here?

> [@talqadi7](#):
>
> "The specifications statement is identical to the Source\_ID device.  
> RE: S123039  
> Index\_ID S123039  
> Source\_ID Electrical Appliance S120492  
> Dated October 6, 2016  
> Received October 7, 2016"

If it is multiline, then you would need to convert it to a single line using the multiline filter, and then apply grok. Your grok pattern looks perfectly fine and as tested in the Kibana grok debugger, it works perfectly fine, with a minor tweak. But I would be making some assumptions about the message structure here or more precisely the field placement

```auto
%{GREEDYDATA} Index_ID %{GREEDYDATA} Source_ID%{GREEDYDATA}[sS]%{NUMBER:number} %{GREEDYDATA}

```

* * *

Let's take the following log message as example:

```auto
{
  'Applicant_field': 'bob',
  'Index_ID': 'kimchi', 
  'message': 'The specifications statement is identical to the Source_ID device. RE: S123039 Index_ID S123039 Source_ID Electrical Appliance S120492 Dated October 6, 2016 Received October 7, 2016'
}

```

The logstash filter that would parse this info would look something as follows:

```auto
filter {
  json {
    source => "message"
    target => "parsed"
  }
  grok {
    match => { "[parsed][message]" => "%{GREEDYDATA} Index_ID %{GREEDYDATA} Source_ID%{GREEDYDATA}[sS]%{NUMBER:number} %{GREEDYDATA}"}
  }
}

```

In case this does not work, you could have conditionals, where you can check for `_grokparsefailure` in `tags` and then apply a second grok filter. Hope this helps.

PS: Please use formatting to make your posts a little more readable. 🙂

Cheers!

---

<div class="post-metadata">

**Author:** ![talqadi7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talqadi7/32/43703_2.png) [@talqadi7](https://discuss.elastic.co/u/talqadi7)\
**Post date:** [June 23, 2019, 5:14am UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/5 "2019-06-23T05:14:52Z")

</div>

Thank you so much for your help! Please let me know if I'm being too troublesome.  
This wont work because sometimes the message field may look like the following:

> {  
> "The specifications statement is identical to the Source\_ID device.  
> RE: S123039  
> Index\_ID S123039  
> Dated October 6, 2016  
> Received October 7, 2016"  
> ......_TEXT_......  
> Source\_ID Electrical Appliance S120492  
> }

So with this application, 50% of the cases, where the word 'source\_id' was present in the _TEXT_ part, the source\_id was extracted as the index\_id, because the id itself would also be mentioned in the text. Do you have an idea on how I could fix this? A simple conditional within grok would work, where it ignores an ID == to index\_id.

I've been trying to work on this problem; however, can't find a way forward. Any help would be appreciated. Thank you so much.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [June 23, 2019, 6:48pm UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/6 "2019-06-23T18:48:50Z")

</div>

> [@talqadi7](#):
>
> Do you have an idea on how I could fix this? A simple conditional within grok would work, where it ignores an ID == to index\_id.

Yes. This too is possible. Use conditionals:

> **[Accessing event data and fields | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals)**

```auto
if [parsed][index_id] == [parsed][source_id]
{
  grok {
    remove_field => ["field names you don't need"]
    match => { "[parsed][message]" => "%{GREEDYDATA} Index_ID %{GREEDYDATA} Source_ID%{GREEDYDATA}[sS]%{NUMBER:number} %{GREEDYDATA} Source_ID%{GREEDYDATA}[sS]%{NUMBER:number}" }
  }
}

```

Notice the use of two GREEDYDATA to identify the second instance of Source\_ID field.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 23, 2019, 7:07pm UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/7 "2019-06-23T19:07:07Z")

</div>

Having multiple GREEDYDATA fields can be really expensive since it can lead to backtracking and even timeouts. I would anchor the patterns to known parts of the message

```
    grok { match => { message => ["^Index_ID %{WORD:Index_ID}$"] } }
    grok { match => { message => ["^Source_ID %{DATA} [Ss]%{INT:Source_ID}$" ] } }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2019, 7:07pm UTC](https://discuss.elastic.co/t/grok-trying-to-continue-with-pattern-based-on-match/186185/8 "2019-07-21T19:07:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.