# Can I add (arbitrary) new fields from inside a grok pattern?

**URL:** <https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497>\
**Category:** Logstash\
**Created:** [February 25, 2017, 8:12pm UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497 "2017-02-25T20:12:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nick-george](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nick-george/32/23318_2.png) [@nick-george](https://discuss.elastic.co/u/nick-george)\
**Post date:** [February 25, 2017, 8:12pm UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/1 "2017-02-25T20:12:55Z")

</div>

Hi,

I have been making use of freely available logstash configs and patterns, like this ([https://github.com/nxhack/logstash/blob/master/patterns/sshd](https://github.com/nxhack/logstash/blob/master/patterns/sshd)) awesome one for SSHD. I've copied a small section of the file for illustrative purposes below:

```
# Normal
SSHD_SUCCESS (?<sshd_result>Accepted) %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}
SSHD_DISCONNECT Received disconnect from %{IP:sshd_client_ip}: (?<sshd_disconnect_code>\d+): %{GREEDYDATA:sshd_disconnect_reason}
SSHD_CONN_CLOSE Connection closed by %{IP:sshd_client_ip}
SSHD_SESSION_OPEN pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\)
SSHD_SESSION_CLOSE pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user}
SSHD_NORMAL_LOG %{SSHD_SUCCESS}|%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE}|%{SSHD_SESSION_OPEN}|%{SSHD_SESSION_CLOSE}

```

What makes this so elegant is that we can have a single grok filter in logstash but match lots of potential log lines. This makes for a VERY clean logstash config.

Where this design lets us down is capturing the meaning/result of each of the log lines. You can see that there's an attempt in the top pattern to set the sshd\_result field, and we could potentially add something similar to the rest of the patterns in the file. However, a much cleaner way would be if we could add arbitrary new fields from within the grok pattern itself. Is this already possible, and I just don't know how to do it?

An example of what I'm talking about is that if the SSHD\_SUCCESS pattern matched, I'd like to create a field called 'action' with a value of 'login', and another field called 'result' with a value of 'success'. I'd like to do this (or similar) for all the patterns in the file. As far as I know, doing this at the moment requires creating a separate GROK for every single pattern in the file, which blows out the size of the code big time.

I think the 'design pattern' above is the cleanest way to do grokking en masse. I'd really like to be able to use it all the time, but this apparent limitation makes is very difficult. Any suggestions would be gratefully received!

Regards,  
Nick

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 27, 2017, 6:48am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/2 "2017-02-27T06:48:50Z")

</div>

I'm afraid you need multiple grok filters, each with an `add_field` option to add the fields you want.

---

<div class="post-metadata">

**Author:** ![nick-george](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nick-george/32/23318_2.png) [@nick-george](https://discuss.elastic.co/u/nick-george)\
**Post date:** [March 2, 2017, 2:50am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/3 "2017-03-02T02:50:55Z")

</div>

Hi Magnus,

Is there any chance of such a feature being implemented? It would open up a whole new approach to doing large amounts of grokking, while allowing for much smaller (less error prone) logstash configs.

Cheers,  
Nick

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 2, 2017, 6:58am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/4 "2017-03-02T06:58:50Z")

</div>

While convenient in special cases I'm not sure I'd be in favor of such a feature, but it's not up to me. I suggest you file a GitHub issue: [https://github.com/logstash-plugins/logstash-filter-grok/issues/new](https://github.com/logstash-plugins/logstash-filter-grok/issues/new)

---

<div class="post-metadata">

**Author:** ![nick-george](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nick-george/32/23318_2.png) [@nick-george](https://discuss.elastic.co/u/nick-george)\
**Post date:** [March 2, 2017, 7:43am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/5 "2017-03-02T07:43:54Z")

</div>

Thanks Magnus,

Will do.

Nick

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2017, 7:43am UTC](https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497/6 "2017-03-30T07:43:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.