# Need support in parsing Syslog messages

**URL:** <https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149>\
**Category:** Logstash\
**Created:** [August 10, 2015, 6:22pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149 "2015-08-10T18:22:51Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 10, 2015, 6:22pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/1 "2015-08-10T18:22:51Z")

</div>

I want to parse the below syslog message.

Jul 23 11:39:11 cheerwine CEF: 0|OWASP|appsensor|1.0|IE1|Input  
Validation|7|cat=attack\_detection deviceExternalId=localhostme suser=bob  
cn1Label=thresholdCount cn1=3 cn2Label=intervalDuration cn2=5  
cs1Label=intervalUnit cs1=minutes

I want to extract as below  
"Detection\_point\_Category"= "Input Validation", "Detection\_point\_label"="IE1" and  
values of "cat", "deviceExternalID"and "suser".

I know we need to use KV filter for parsing. I put lots of effort on making it work but I was unsuccessful.Looking forward for the help from Elastic community.

The code I am currently working on is below:

input {  
tcp {  
port =\> 5000  
type =\> syslog  
}  
}

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGBASE}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}

syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}

}  
}

output {  
elasticsearch { host =\> localhost  
index =\> "dashboard"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 10, 2015, 7:22pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/2 "2015-08-10T19:22:09Z")

</div>

Start with a grok filter to separate the pieces that are suitable for parsing with the kv filter, then apply kv to the rest. Something like this (untested) should get you started:

```
filter {
  grok {
    match => [
      "message",
      "... 0\|OWASP\|appsensor\|1\.0\|%{WORD:Detection_point_label}\|%{WORDDetection_point_Category}\|7\|%{GREEDYDATA:keyvalues}"
    ]
  }
  kv {
    source => "keyvalues"
    remove_field => ["keyvalues"]
  }
}

```

I've obviously omitted the beginning of the string in the grok expression (you should be able to use `%{SYSLOGBASE}` there) and you'll want to replace the hardcoded field values ("appsensor", "7", ...) with something that matches the kind of data you might get there. Or, you could make it match any character that isn't a pipe character with `[^|]+`.

You could also use a csv filter to split the pipe-separated pieces. That might actually be more elegant.

---

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 11, 2015, 6:39am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/3 "2015-08-11T06:39:48Z")

</div>

I have tried the below two types of codes but the both resulted in grok filter failure.

Code-1:  
match =\> [  
"message",  
"... 0|OWASP|appsensor|1.0|%{WORD:Detection\_point\_label}|%{WORDDetection\_point\_Category}|7|%{GREEDYDATA:keyvalues}"  
]

Code-2:  
match =\> [  
"message",  
"%{SYSLOGBASE}|OWASP|appsensor|1.0|%{WORD:Detection\_point\_label}|%{WORDDetection\_point\_Category}|7|%{GREEDYDATA:keyvalues}"  
]

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2015, 6:45am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/4 "2015-08-11T06:45:22Z")

</div>

Well, the first one obviously failed since my example was pseudo code. In the second attempt you've introduced SYSLOGBASE, which is defined like this:

```
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

```

Assuming that this matches your log (I think it does but haven't looked closely) then at the very least you're missing a space and the initial zero after SYSLOGBASE. This should be closer to working:

```
%{SYSLOGBASE} 0\|OWASP\|...
```

---

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 11, 2015, 8:27am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/5 "2015-08-11T08:27:04Z")

</div>

The below is the complete code I was using to extract the required data from the given syslog. I had made the necessary changes suggested by you but still it resulting in grok filter failure

Syslog:  
Jul 23 11:39:11 cheerwine CEF: 0|OWASP|appsensor|1.0|IE1|Input  
Validation|7|cat=attack\_detection deviceExternalId=localhostme suser=bob  
cn1Label=thresholdCount cn1=3 cn2Label=intervalDuration cn2=5  
cs1Label=intervalUnit cs1=minutes

Updated logstash config file:

input {  
tcp {  
port =\> 5000  
type =\> syslog  
}  
}

filter {  
if [type] == "syslog" {  
grok {  
match =\> [  
"message",  
"%{SYSLOGBASE} 0|OWASP|appsensor|1.0|%{WORD:Detection\_point\_label}|%{WORDDetection\_point\_Category}|7|%{GREEDYDATA:keyvalues}"  
]  
}  
kv {  
source =\> "keyvalues"  
remove\_field =\> ["keyvalues"]  
}

}  
}

output {  
elasticsearch { host =\> localhost  
index =\> "dashboard"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2015, 8:39am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/6 "2015-08-11T08:39:46Z")

</div>

> ```
> ... WORDDetection_point_Category ...
> 
> ```

This should be `WORD:Detection_point_Category`. That was a typo from what I wrote last night, sorry.

---

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 11, 2015, 8:40am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/7 "2015-08-11T08:40:39Z")

</div>

I corrected and executed it long back... But still the error persists

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2015, 8:51am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/8 "2015-08-11T08:51:15Z")

</div>

Replace `%{WORD:Detection_point_Category}` with `(?<Detection_point_Category>[^|]+)`.

---

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 11, 2015, 1:15pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/9 "2015-08-11T13:15:12Z")

</div>

Output after making suggested changes:

Syslog used:  
Jul 23 11:39:11 cheerwine CEF: 0|OWASP|appsensor|1.0|IE1|Input  
Validation|7|cat=attack\_detection deviceExternalId=localhostme suser=bob  
cn1Label=thresholdCount cn1=3 cn2Label=intervalDuration cn2=5  
cs1Label=intervalUnit cs1=minutes

Output generated:  
{  
"message" =\> "Jul 23 11:39:11 cheerwine CEF: 0|OWASP|appsensor|1.0|IE1|Input\r",  
"@version" =\> "1",  
"@timestamp" =\> "2015-08-11T13:10:11.723Z",  
"host" =\> "127.0.0.1",  
"type" =\> "syslog",  
"tags" =\> [  
[0] "\_grokparsefailure"  
]  
}  
{  
"message" =\> "Validation|7|cat=attack\_detection deviceExternalId=localhostme suser=bob\r",  
"@version" =\> "1",  
"@timestamp" =\> "2015-08-11T13:10:11.726Z",  
"host" =\> "127.0.0.1",  
"type" =\> "syslog",  
"tags" =\> [  
[0] "\_grokparsefailure"  
]  
}  
{  
"message" =\> "cn1Label=thresholdCount cn1=3 cn2Label=intervalDuration cn2=5\r",  
"@version" =\> "1",  
"@timestamp" =\> "2015-08-11T13:10:11.726Z",  
"host" =\> "127.0.0.1",  
"type" =\> "syslog",  
"tags" =\> [  
[0] "\_grokparsefailure"  
]  
}

Filter block code:  
filter {  
grok {  
match =\> ["message","%{SYSLOGBASE} 0|OWASP|appsensor|1.0|%{WORD:Detection\_point\_label}|(?\<Detection\_point\_Category\>[^|]+)|7|%{GREEDYDATA:keyvalues}"]  
}

kv {  
source =\> "keyvalues"  
remove\_field =\> ["keyvalues"]  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2015, 1:21pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/10 "2015-08-11T13:21:19Z")

</div>

What, do your syslog messages span multiple lines? Are you still using a TCP input?

---

<div class="post-metadata">

**Author:** ![Sumanth\_Damarla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sumanth_damarla/32/3802_2.png) [@Sumanth\_Damarla](https://discuss.elastic.co/u/Sumanth_Damarla)\
**Post date:** [August 11, 2015, 1:27pm UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/11 "2015-08-11T13:27:17Z")

</div>

I got it Magnus, Thanks for ur support. It means a lot to me 😄

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:32am UTC](https://discuss.elastic.co/t/need-support-in-parsing-syslog-messages/27149/12 "2017-07-06T05:32:21Z")

</div>