# Extracting the message and adding new fields dynamically

**URL:** <https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412>\
**Category:** Logstash\
**Created:** [April 2, 2018, 10:04am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412 "2018-04-02T10:04:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Santhosh\_ic](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@Santhosh\_ic](https://discuss.elastic.co/u/Santhosh_ic)\
**Post date:** [April 2, 2018, 10:04am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/1 "2018-04-02T10:04:11Z")

</div>

Hi,

I have below mentioned log events. From the last part, slog\_message, "GREEDYDATA" in the grok, I would like to match again on that and extract various data out of it. For example, device id's, usernames etc. How can that be achieved? I want to create Kibana dashboards grouped by the device ids or usernames etc.

==========Log events=================  
2018/02/09 09:21:08.800 CONSOLE 7aab3115-03e8-42d4-94ed-a7af754ffa6e [0000000-0000000] (12) Info a.b.c.d.e.listener 0 Agent Profiles To Install For Device ID 2  
2018/03/30 09:47:35.081 CONSOLE 95e3c58a-07bc-4354-8f71-0f3d133daa43 [0000000-0000000] (8) Info a.b.c.d.e.LoginController Invalid user for console login: adminafaff  
2018/03/30 09:47:35.081 CONSOLE 597bd229-3f41-49ac-9e44-6559dcb63a81 [0000000-0000000] (10) Info a.b.c.Web.d.e.authenticate User is not logged in, redirecting to login page  
2018/03/30 09:47:35.128 CONSOLE 95e3c58a-07bc-4354-8f71-0f3d133daa43 [0000000-0000000] (8) Info a.b.c.Web.d.e.Controllers.LoginController New invalid user: adminafaff

==========Configuration=================  
input {  
beats {  
port =\> 5044  
}  
}

filter {  
grok {  
match =\> {"message" =\> "(?\<slog\_date\>^\d{4}/\d{2}/\d{2})\s{1,}%{TIME: slog\_time\_stamp}\s{1,}%{USERNAME:slog\_host}\s{1,}%{UUID:slog\_uuid}\s{1,}(?\<slog\_thread-id\>[[0-9-]+])\s{1,}((?\<slog\_pid\>[0-9]+))\s{1,}%{LOGLEVEL:slog\_level}\s{1,}%{GREEDYDATA:slog\_message}"}  
}

//Something of this sort.  
if [slog\_message] =~ "New invalid user:" {  
//extract the username from the slog\_message  
match =\> {slog\_message =\> "^[a-zA-Z.]+\s{1,}New invalid user:\s{1,}(?\<invalid\_user\>[a-zA-Z]+)\s{0,}$"}  
}  
if [slog\_message] =~ "Device ID" {  
//extract device id "2"  
}  
if [slog\_message] =~ "Invalid user for console login:" {  
//extract the username  
}  
}

output {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 3, 2018, 6:29pm UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/2 "2018-04-03T18:29:30Z")

</div>

> //Something of this sort.  
> if [slog\_message] =~ "New invalid user:" {  
> //extract the username from the slog\_message  
> match =\> {slog\_message =\> "\[1\]+\s{1,}New invalid user:\s{1,}(?\<invalid\_user\>[a-zA-Z]+)\s{0,}$"}  
> }

You're close:

```plaintext
//Something of this sort.
if [slog_message] =~ "New invalid user:" {
  grok {
    //extract the username from the slog_message
    match => {slog_message => "^[a-zA-Z.]+\s{1,}New invalid user:\s{1,}(?<invalid_user>[a-zA-Z]+)\s{0,}$"}
  }
}

```

* * *

1. a-zA-Z.

---

<div class="post-metadata">

**Author:** ![Santhosh\_ic](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@Santhosh\_ic](https://discuss.elastic.co/u/Santhosh_ic)\
**Post date:** [April 11, 2018, 5:39am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/3 "2018-04-11T05:39:19Z")

</div>

Thank you @magnusbaeck for the quick response. That worked.

Was my approach the efficient or the right way to do? Are there any other better ways of achieving it?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 11, 2018, 5:59am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/4 "2018-04-11T05:59:29Z")

</div>

You could've just skipped the conditional and tried the grok filter with `tag_on_failure` set to an empty array.

---

<div class="post-metadata">

**Author:** ![Santhosh\_ic](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@Santhosh\_ic](https://discuss.elastic.co/u/Santhosh_ic)\
**Post date:** [April 23, 2018, 10:08am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/5 "2018-04-23T10:08:49Z")

</div>

@magnusbaeck can this step be done with elastic search instead of doing this in the log stash phase? If we can, then how to do it with elastic search?  
Which is more efficient? groking in log-stash or doing it with elastic search?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 23, 2018, 10:35am UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/6 "2018-04-23T10:35:24Z")

</div>

You should be able to set up an Elasticsearch ingest pipeline to perform the same thing. I don't know which options performs best.

---

<div class="post-metadata">

**Author:** ![Santhosh\_ic](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@Santhosh\_ic](https://discuss.elastic.co/u/Santhosh_ic)\
**Post date:** [April 23, 2018, 2:44pm UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/7 "2018-04-23T14:44:27Z")

</div>

Thank you for a quick response.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2018, 2:50pm UTC](https://discuss.elastic.co/t/extracting-the-message-and-adding-new-fields-dynamically/126412/8 "2018-05-21T14:50:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.