# \[SOLVED\] Syslog filter additional filtering

**URL:** <https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526>\
**Category:** Logstash\
**Created:** [November 9, 2016, 4:08pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526 "2016-11-09T16:08:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![aeva\_assured](https://avatars.discourse-cdn.com/v4/letter/a/35a633/32.png) [@aeva\_assured](https://discuss.elastic.co/u/aeva_assured)\
**Post date:** [November 9, 2016, 4:08pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/1 "2016-11-09T16:08:26Z")

</div>

Hello all. I'm very new to logstash and I'm trying to add additional fields to an existing syslog filter. My base filter is:

`
#Rubrik Syslog Input
input {
  tcp {
     host => "192.168.1.180"
     port => 1514
  }
}
filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "SQL" in [syslog5424_msg] {
grok {
add_field => ["SQL","SQL"]
}
}
}
output {
stdout { codec => rubydebug }
}
`

The output I get is:  
`
{
"message" => "<134>1 2016-11-09T15:13:37.434Z RVM155S027019 Rubrik - - [mdc@18060 ndc="LITE_REFRESH_METADATA_f3a43436-442f-4c0a-992d-3a1f5b851692:::2178" pid="10163"] (eventId=1478704417420-65f6e3fc-35a3-47e3-ace2-618f99085326 eventSeriesId=1f62f2dc-e299-477b-9d4b-8bd06d2ec603 objectId=f3a43436-442f-4c0a-992d-3a1f5b851692 status=Success) Finished refreshing vCenter '192.168.1.4'",
"@version" => "1",
"@timestamp" => "2016-11-09T15:09:53.377Z",
"host" => "192.168.1.103",
"port" => 40134,
"syslog5424_pri" => "134",
"syslog5424_ver" => "1",
"syslog5424_ts" => "2016-11-09T15:13:37.434Z",
"syslog5424_host" => "RVM155S027019",
"syslog5424_app" => "Rubrik",
"syslog5424_sd" => "[mdc@18060 ndc="LITE_REFRESH_METADATA_f3a43436-442f-4c0a-992d-3a1f5b851692:::2178" pid="10163"]",
"syslog5424_msg" => "(eventId=1478704417420-65f6e3fc-35a3-47e3-ace2-618f99085326 eventSeriesId=1f62f2dc-e299-477b-9d4b-8bd06d2ec603 objectId=f3a43436-442f-4c0a-992d-3a1f5b851692 status=Success) Finished refreshing vCenter '192.168.1.4'"
}
`

I'd like to further part "syslog5424\_msg" based on values that I find in there, such as SQL or vCenter. I know that the if statement is correctly being met positive because with non-SQL messages I get no parsing errors. However, I can't figure out for the of me how to sub-parse syslog5424\_msg into smaller pieces.

Ideally I want:  
eventId=  
eventSeriesId  
objectId=  
status=  
hostname=  
actiontype="refreshing vCenter"

I've been playing a lot with grokdebug,[herokuapp.com](http://herokuapp.com) and looking at pattern examples until my eyes are crossed. I'm missing something totally basic and need some guidance on adding custom patterns either within my conf file, or as a separate patterns file.

Ultimately I expect I'm going to need to build a new plugin as I'll have a lot of parsing as the messages tend to vary. But baby steps!

Thank you all in advance.

---

<div class="post-metadata">

**Author:** ![aeva\_assured](https://avatars.discourse-cdn.com/v4/letter/a/35a633/32.png) [@aeva\_assured](https://discuss.elastic.co/u/aeva_assured)\
**Post date:** [November 9, 2016, 5:09pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/2 "2016-11-09T17:09:19Z")

</div>

I'm making progress, I am able to pull part of the eventId:

`
#Syslog Input
input {
  tcp {
     host => "192.168.1.180"
     port => 1514
  }
}
filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "MSSQL_DB_BACKUP" in [syslog5424_sd] {
grok {
match => {
"syslog5424_msg" => ["(eventId=%{NUMBER:eventId}-"]
}
}
}
}
output {
stdout { codec => rubydebug }
}
`

with result:

`
{
            "message" => "<134>1 2016-11-09T17:06:56.858Z RVM154S014332 Rubrik - - [mdc@18060 ndc=\"MSSQL_DB_BACKUP_5282f305-967f-4d47-bd2f-a6a0a3f3ea0a:::526\" pid=\"10863\"] (eventId=1478711216846-2c5b4ce5-30fb-4b21-878f-6244d2b5c66d eventSeriesId=16a305d9-1f7a-4e80-81db-ad968beef4af objectId=b74f3482-5a49-4334-b7f8-2b47611cafdb@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed backup of Microsoft SQL Database 'MyTestDB' from 'ADP-MON01'",
           "@version" => "1",
         "@timestamp" => "2016-11-09T17:03:12.638Z",
               "host" => "192.168.1.102",
               "port" => 34825,
     "syslog5424_pri" => "134",
     "syslog5424_ver" => "1",
      "syslog5424_ts" => "2016-11-09T17:06:56.858Z",
    "syslog5424_host" => "RVM154S014332",
     "syslog5424_app" => "Rubrik",
      "syslog5424_sd" => "[mdc@18060 ndc=\"MSSQL_DB_BACKUP_5282f305-967f-4d47-bd2f-a6a0a3f3ea0a:::526\" pid=\"10863\"]",
     "syslog5424_msg" => "(eventId=1478711216846-2c5b4ce5-30fb-4b21-878f-6244d2b5c66d eventSeriesId=16a305d9-1f7a-4e80-81db-ad968beef4af objectId=b74f3482-5a49-4334-b7f8-2b47611cafdb@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed backup of Microsoft SQL Database 'MyTestDB' from 'ADP-MON01'",
            "eventId" => "1478711216846"
}
`

But I'm still missing some fundamentals. I need all the pieces until the next white space to have the full ID and 'GREEDYDATA' gives me too much, 'DATA' gives me nothing, and trying to put in a custom pattern has create a lot of swearing and parse failures.

---

<div class="post-metadata">

**Author:** ![aeva\_assured](https://avatars.discourse-cdn.com/v4/letter/a/35a633/32.png) [@aeva\_assured](https://discuss.elastic.co/u/aeva_assured)\
**Post date:** [November 9, 2016, 6:11pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/3 "2016-11-09T18:11:44Z")

</div>

OK I've made some great progress. I'm capturing properly, I am confident in my parse patterns, but I'm not quite getting what I want in the output, here is my config:

`
 Syslog Input
input {
  tcp {
     host => "192.168.1.180"
     port => 1514
  }
}
filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "MSSQL_DB_BACKUP" in [syslog5424_sd] {
grok {
match => {
"syslog5424_msg" => ["(eventId=%{DATA:eventId} ev" , "Database '%{DATA:dbname}'" , "from '%{DATA:bkuptarget}'"]
}
}
}
}
output {
stdout { codec => rubydebug }
}
`

Here is the output:  
`
{
"message" => "<134>1 2016-11-09T18:06:22.303Z RVM154S014332 Rubrik - - [mdc@18060 ndc="MSSQL_DB_BACKUP_e2560421-0d99-4966-bfee-b53f809cca4f_ea9696ef-e71c-4494-ae46-a8c5126e4b7e:::0" pid="10863"] (eventId=1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e eventSeriesId=1a896234-e5d6-44b6-bb50-687e11e531d8 objectId=87fcda14-e7fb-43cb-9428-75dbf9b7124c@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed manual backup of Microsoft SQL Database 'master' from 'ADP-MON01'",
"@version" => "1",
"@timestamp" => "2016-11-09T18:02:37.997Z",
"host" => "192.168.1.102",
"port" => 49122,
"syslog5424_pri" => "134",
"syslog5424_ver" => "1",
"syslog5424_ts" => "2016-11-09T18:06:22.303Z",
"syslog5424_host" => "RVM154S014332",
"syslog5424_app" => "Rubrik",
"syslog5424_sd" => "[mdc@18060 ndc="MSSQL_DB_BACKUP_e2560421-0d99-4966-bfee-b53f809cca4f_ea9696ef-e71c-4494-ae46-a8c5126e4b7e:::0" pid="10863"]",
"syslog5424_msg" => "(eventId=1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e eventSeriesId=1a896234-e5d6-44b6-bb50-687e11e531d8 objectId=87fcda14-e7fb-43cb-9428-75dbf9b7124c@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed manual backup of Microsoft SQL Database 'master' from 'ADP-MON01'",
"eventId" => "1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e"
}
`

I'm now getting the full eventId, but my other pattersn aren't getting matched as I would expect. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 10, 2016, 6:52am UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/4 "2016-11-10T06:52:18Z")

</div>

That's because you're listing multiple patterns in the same grok filter and it'll stop after the first match. Use a single expression:

```
\(eventId=%{NOTSPACE:eventId} .* Database '(?<dbname>[^']+)' from '(?<bkuptarget>[^']+)'

```

}

---

<div class="post-metadata">

**Author:** ![aeva\_assured](https://avatars.discourse-cdn.com/v4/letter/a/35a633/32.png) [@aeva\_assured](https://discuss.elastic.co/u/aeva_assured)\
**Post date:** [November 10, 2016, 2:51pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/5 "2016-11-10T14:51:01Z")

</div>

Awesome. Does that mean that the functionality of:

match =\>  
"item" =\> ["pattern1" , "pattern2" , "pattern3"]

is to find the FIRST matching pattern and then output? So if none of them match I'll get a parsefailure and if any one of them matches the first match will be recorded?

Really appreciate the help, has made a very big difference.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 10, 2016, 3:05pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/6 "2016-11-10T15:05:40Z")

</div>

> is to find the FIRST matching pattern and then output? So if none of them match I'll get a parsefailure and if any one of them matches the first match will be recorded?

Exactly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2016, 3:06pm UTC](https://discuss.elastic.co/t/solved-syslog-filter-additional-filtering/65526/7 "2016-12-08T15:06:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.