# Message field within message

**URL:** <https://discuss.elastic.co/t/message-field-within-message/58051>\
**Category:** Logstash\
**Created:** [August 15, 2016, 6:02pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051 "2016-08-15T18:02:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 15, 2016, 6:02pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051/1 "2016-08-15T18:02:30Z")

</div>

Hi All,

Raw SYSLOG event:

Jul 21 01:19:57.58 172.20.20.100 date=2016-07-20 time=20:12:25 timezone="UTC" device\_name="CRiV-1C" device\_id=1234-567 log\_id=062009517502 log\_type="Event" log\_component="GUI" log\_subtype="Admin" status="Successful" priority=Notice user\_name="admin" src\_ip=10.0.0.26 message="Appliance Access Settings were changed by 'admin' from '10.0.0.26' using 'GUI'"

Problem Statement:

As per my knowledge Logstash encapsulate entire syslog event into a payload or record called "message". This make sense to me but in my case this is conflicting with "message" key which already exist in the event.

My requirement:

Rename "message" key (Not the entire payload) to "dmsg".

I tried grok filter to rename this, that works perfectly fine. But it creates mess if message key in some syslog event doesn't exist and it simply copy entire syslog event into dmsg. I would like to copy message key into dmsg only when that exist in the syslog event.

I would appreciate if someone could help me.

Regards,

Ajay

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 15, 2016, 7:53pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051/2 "2016-08-15T19:53:32Z")

</div>

So do you want

```
message: Jul 21 01:19:57.58 172.20.20.100 ...
dmsg: Appliance Access Settings ...

```

or

```
message: Appliance Access Settings ...
dmsg: Jul 21 01:19:57.58 172.20.20.100 ...
```

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 6:40am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/3 "2016-08-16T06:40:08Z")

</div>

> [@magnusbaeck](#):
>
> message: Jul 21 01:19:57.58 172.20.20.100 ...  
> dmsg: Appliance Access Settings ...

Yes this is what I would like to achieve.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 7:20am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/4 "2016-08-16T07:20:40Z")

</div>

And you use a grok filter to parse the event and extract the `log_type`, `log_component`, ..., `message` fields? If so, why not just extract the message to `dmsg` instead of `message`. If you show your current configuration it'll be easier to help.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:03am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/5 "2016-08-16T09:03:43Z")

</div>

I am not sure how to extract a field named as "message" from grok, I would appreciate if you could guide me or redirect me to some article or doc.

Extracting means old message key won't be part of event and a new key dmsg would be in the event. Is this a correct understanding?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 9:09am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/6 "2016-08-16T09:09:16Z")

</div>

I repeat: If you show your current configuration it'll be easier to help.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:15am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/7 "2016-08-16T09:15:07Z")

</div>

Here is my current configuration for testing purpose:

filter {  
if ("cloned" not in [tags]) {  
clone {  
clones =\> ["cyberoam.raw"]  
add\_tag =\> ["cloned"]  
}  
}

```
    if [type] == "cyberoam.input" {

            kv { }

            grok {
                    patterns_dir => ["/etc/logstash/patterns"]
                    match => ["message", "%{CRv11}"]
            }

            syslog_pri { }

            mutate {
                    rename => ["src_ip" , "srcip"]
                    rename => ["dst_ip" , "dstip"]
                    rename => ["message", "dmsg"]
                    rename => ["device_name", "devname"]
                    rename => ["device_id", "devid"]
			}
	}

```

}

output {

if [type] == "cyberoam.input" {  
stdout {  
codec =\> rubydebug { metadata =\> true }  
}

```
    stdout { }

```

}

if [type] == "cyberoam.raw" { stdout { } }

}

In above, I am just renaming the field message to dmsg as it was creating problem in ES index mapping.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 9:16am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/8 "2016-08-16T09:16:00Z")

</div>

And what's the definition of the CRv11 grok pattern?

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:16am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/9 "2016-08-16T09:16:45Z")

</div>

CRv11 ^\<%{NONNEGINT:syslog\_priority}\>date=%{CRDATE:date} time=%{TIME:time} timezone=%{TZ:timezone} device\_name=%{WORD:device\_name} device\_id=%{WORD:device\_id} log\_id=%{NUMBER:log\_id} log\_type=%{WORD:log\_type} log\_component=%{WORD:log\_component} log\_subtype=%{WORD:log\_subtype} status=%{WORD:status} priority=%{WORD:priority} fw\_rule\_id=%{NUMBER:fw\_rule\_id} user\_name=%{WORD:user\_name} user\_gp=%{WORD:user\_gp} iap=%{NUMBER:iap} category=%{WORD:category} category\_type=%{WORD:category\_type} url=%{WORD:url} contenttype=%{WORD:contenttype} override\_token=%{WORD:override\_token} httpresponsecode=%{WORD:httpresponsecode} sourceip=%{IP:src\_ip} dst\_ip=%{IP:dst\_ip} protocol=%{NUMBER:protocol} src\_port=%{NUMBER:src\_port} dst\_port=%{NUMBER:dst\_port} sent\_bytes=%{NUMBER:sent\_bytes} recv\_bytes=%{NUMBER:recv\_bytes} domain=%{WORD:domain}

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:17am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/10 "2016-08-16T09:17:25Z")

</div>

Above pattern doesn't contain "message" field, as I have not yet completed writing all fields in the pattern.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 9:21am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/11 "2016-08-16T09:21:07Z")

</div>

Okay, but when you do complete the pattern you could say

```
message=%{SOMEGROKPATTERN:dmsg}

```

instead of

```
message=%{SOMEGROKPATTERN:message}

```

to capture directly to the `dmsg` field.

But really, you should use a kv filter for parsing these key/value pairs instead of writing a really long grok pattern that'll break for even small changes in the log data.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:22am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/12 "2016-08-16T09:22:26Z")

</div>

I am not sure how to write KV filters. If you could just give me a hint it would be really great.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 9:25am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/13 "2016-08-16T09:25:03Z")

</div>

Use a grok filter to extract `syslog_priority` and put the rest of the string into a field that you feed a kv filter. There are tons of kv filter examples available online (e.g. in this forum) so I won't write an example for that.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 9:27am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/14 "2016-08-16T09:27:03Z")

</div>

Ok, let me learn and try that. Thank you for your prompt responses.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 16, 2016, 2:04pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051/15 "2016-08-16T14:04:58Z")

</div>

I am facing problem with KV:

Below is sample event:

2016-08-15T12:43:04.478Z 103.240.35.216 \<190\>date=2016-08-15 time=18:13:16 timezone="IST" device\_name="CR25iNG" device\_id=C2222-123 log\_id=010302602002 log\_type="Firewall" log\_component="Appliance Access" log\_subtype="Denied" status="Deny" priority=Information duration=0 fw\_rule\_id=0 user\_name="" user\_gp="" iap=0 ips\_policy\_id=0 appfilter\_policy\_id=0 application="" application\_risk=0 application\_technology="" application\_category="" in\_interface="PortA" out\_interface="" src\_mac=44:d9:e7:ba:5b:6c src\_ip=172.16.16.19 src\_country\_code= dst\_ip=255.255.255.255 dst\_country\_code= protocol="UDP" src\_port=45541 dst\_port=10001 sent\_pkts=0 recv\_pkts=0 sent\_bytes=0 recv\_bytes=0 tran\_src\_ip= tran\_src\_port=0 tran\_dst\_ip= tran\_dst\_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir\_disp="" connid="" vconnid=""

Below is the KV filter output:

"@version" =\> "1",  
"@timestamp" =\> "2016-08-16T13:48:30.602Z",  
"type" =\> "cyberoam.input",  
"host" =\> "ip-172-31-6-249",  
"time" =\> "18:13:16",  
"timezone" =\> "IST",  
"status" =\> "Deny",  
"priority" =\> "Information",  
"duration" =\> "0",  
"iap" =\> "0",  
"application" =\> "",  
"application\_risk" =\> "0",  
"application\_technology" =\> "",  
"application\_category" =\> "",  
"dst\_country\_code" =\> "protocol=UDP",  
"recv\_pkts" =\> "0",  
"tran\_src\_ip" =\> "tran\_src\_port=0",  
"tran\_dst\_ip" =\> "tran\_dst\_port=0",  
"srczonetype" =\> "",  
"srczone" =\> "",  
"dstzonetype" =\> "",  
"dstzone" =\> "",  
"dir\_disp" =\> "",  
"syslog\_severity\_code" =\> 5,  
"syslog\_facility\_code" =\> 1,  
"syslog\_facility" =\> "user-level",  
"syslog\_severity" =\> "notice",  
"date" =\> "2016-08-15",

Problem:

"tran\_src\_ip" =\> "tran\_src\_port=0",  
"tran\_dst\_ip" =\> "tran\_dst\_port=0",

Above is due to empty keys "tran\_src\_ip" and "tran\_dst\_ip".

Could any one please help me to get this fixed?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 4:56pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051/16 "2016-08-16T16:56:24Z")

</div>

Perhaps you can use a mutate filter and its gsub option to replace occurrences of "\w+= " with an empty string.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 16, 2016, 5:05pm UTC](https://discuss.elastic.co/t/message-field-within-message/58051/17 "2016-08-16T17:05:48Z")

</div>

It looks like `tran_scr_ip` and `tran_dst_ip` have no values defined, which causes the odd parsing. You should be able to add a default value, e.g. `""`, by substituting `=\w` with `=""`.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 17, 2016, 6:57am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/18 "2016-08-17T06:57:18Z")

</div>

Hi Both,

I tried various gsub options but somehow it looks like that gsub is not evaluating event.

Below are configs:

```
          mutate {
                    gsub => ['message', '=\w', '=""']
            }

          mutate {
                    gsub => ['message', '=\s', '=""']
            }

          mutate {
                    gsub => ['message', '\w+=', '""']
            }

```

Please guide.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2016, 7:05am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/19 "2016-08-17T07:05:41Z")

</div>

Why not try the suggestion I made?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 17, 2016, 7:11am UTC](https://discuss.elastic.co/t/message-field-within-message/58051/20 "2016-08-17T07:11:19Z")

</div>

This seems to work for me:

```
mutate {
  gsub => ['message', '= ', '="" ']
}
```

[Next page](https://discuss.elastic.co/t/message-field-within-message/58051.md?page=2)