# Grok pattern for optional field in ingest pipeline

**URL:** <https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [May 12, 2022, 4:42am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520 "2022-05-12T04:42:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![alfianaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfianaf/32/93192_2.png) [@alfianaf](https://discuss.elastic.co/u/alfianaf)\
**Post date:** [May 12, 2022, 4:42am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520/1 "2022-05-12T04:42:25Z")

</div>

Hello, I'm working with ingest pipeline and grok processor right now.  
I need to filter log message with grok processor but there's optional fields,

here's the log example

```auto
[Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms

```

this one has grok pattern

```auto
%{GREEDYDATA:generated_nest} - %{GREEDYDATA:timestamp} \\[%{GREEDYDATA:activity}\\] %{GREEDYDATA:message} %{GREEDYDATA:generated_ms}

```

while the log also has this field

```auto
[Nest] 6448 - 04/26/2022, 3:31:25 PM username | [activity] message | reference +1ms

```

so the grok pattern should be

```auto
%{GREEDYDATA:generated_nest} - %{GREEDYDATA:timestamp} %{GREEDYDATA:username} \\| \\[%{GREEDYDATA:activity}\\] %{GREEDYDATA:message} \\| %{GREEDYDATA:reference} %{GREEDYDATA:generated_ms}

```

from what you can see, the **username** and **reference** field is optional (these two fields is not depend each other),  
is there any way to use one grok pattern, so I don't need to mention each pattern while using "ignore\_failure" on each processor?

any help would be appreciated,

Thanks

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 12, 2022, 5:45am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520/2 "2022-05-12T05:45:13Z")

</div>

Here is your grok:  
`\[%{GREEDYDATA:process}\] %{POSINT:procid} - %{GREEDYDATA:timestamp} \[%{GREEDYDATA:activity}\] %{GREEDYDATA:message} %{GREEDYDATA:generated_ms}`

I would avoid GREEDYDATA since it's slow and add date formatting.

```auto
filter {	
    grok {
	 # [Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms
      match => { "message" => "\[%{DATA:process}\] %{POSINT:procid} - %{DATA:timestamp} \[%{DATA:activity}\] %{DATA:message} %{GREEDYDATA:generated_ms}" }
    }
	
    date {
      match => ["timestamp", "MM/dd/yyyy, H:mm:ss a"]
      timezone => "Asia/Singapore"
     }
}

```

Result:

```auto
{
        "@version" => "1",
       "timestamp" => "04/26/2022, 3:30:03 PM",
        "activity" => "activity",
    "generated_ms" => "+1ms",
      "@timestamp" => 2022-04-25T19:30:03.000Z,
         "process" => "Nest",
          "procid" => "6448",
         "message" => "[Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms"

    ]
}

```

---

<div class="post-metadata">

**Author:** ![alfianaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfianaf/32/93192_2.png) [@alfianaf](https://discuss.elastic.co/u/alfianaf)\
**Post date:** [May 12, 2022, 6:08am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520/3 "2022-05-12T06:08:55Z")

</div>

hello, thanks for the response,  
but it seems like you missed my point, I have two **almost similar** log, there's **username** field and **reference** field which is possible to be included into the log.  
the grok pattern you provided won't matched with my 2nd log,  
is there any way to have one grok pattern with those two different log?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 12, 2022, 8:00am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520/4 "2022-05-12T08:00:19Z")

</div>

You have to handle optional fields or have 2 match patterns. This is 1st option with handling date format.

```auto
    grok {
	  patterns_dir => "./pattern"
      match => { "message" => "\[%{DATA:process}\] %{POSINT:procid} - %{CDATE:timestamp} (%{DATA:username}\s\|\s)?\[%{DATA:activity}\] %{DATA:message} (\|\s%{DATA:ref}\s)?%{GREEDYDATA:generated_ms}" }
    }

```

Put a file in the pattern directory with content:  
`CDATE %{DATE_US}\, %{TIME} %{WORD}`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 9, 2022, 8:00am UTC](https://discuss.elastic.co/t/grok-pattern-for-optional-field-in-ingest-pipeline/304520/5 "2022-06-09T08:00:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.