[SOLVED] Syslog filter additional filtering

Elastic Stack Logstash
1

Hello all. I'm very new to logstash and I'm trying to add additional fields to an existing syslog filter. My base filter is:

#Rubrik Syslog Input input { tcp { host => "192.168.1.180" port => 1514 } }

filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "SQL" in [syslog5424_msg] {
grok {
add_field => ["SQL","SQL"]
}
}
}

output {
stdout { codec => rubydebug }
}

The output I get is:

{
"message" => "<134>1 2016-11-09T15:13:37.434Z RVM155S027019 Rubrik - - [mdc@18060 ndc="LITE_REFRESH_METADATA_f3a43436-442f-4c0a-992d-3a1f5b851692:::2178" pid="10163"] (eventId=1478704417420-65f6e3fc-35a3-47e3-ace2-618f99085326 eventSeriesId=1f62f2dc-e299-477b-9d4b-8bd06d2ec603 objectId=f3a43436-442f-4c0a-992d-3a1f5b851692 status=Success) Finished refreshing vCenter '192.168.1.4'",
"@version" => "1",
"@timestamp" => "2016-11-09T15:09:53.377Z",
"host" => "192.168.1.103",
"port" => 40134,
"syslog5424_pri" => "134",
"syslog5424_ver" => "1",
"syslog5424_ts" => "2016-11-09T15:13:37.434Z",
"syslog5424_host" => "RVM155S027019",
"syslog5424_app" => "Rubrik",
"syslog5424_sd" => "[mdc@18060 ndc="LITE_REFRESH_METADATA_f3a43436-442f-4c0a-992d-3a1f5b851692:::2178" pid="10163"]",
"syslog5424_msg" => "(eventId=1478704417420-65f6e3fc-35a3-47e3-ace2-618f99085326 eventSeriesId=1f62f2dc-e299-477b-9d4b-8bd06d2ec603 objectId=f3a43436-442f-4c0a-992d-3a1f5b851692 status=Success) Finished refreshing vCenter '192.168.1.4'"
}

I'd like to further part "syslog5424_msg" based on values that I find in there, such as SQL or vCenter. I know that the if statement is correctly being met positive because with non-SQL messages I get no parsing errors. However, I can't figure out for the of me how to sub-parse syslog5424_msg into smaller pieces.

Ideally I want:
eventId=
eventSeriesId
objectId=
status=
hostname=
actiontype="refreshing vCenter"

I've been playing a lot with grokdebug,herokuapp.com and looking at pattern examples until my eyes are crossed. I'm missing something totally basic and need some guidance on adding custom patterns either within my conf file, or as a separate patterns file.

Ultimately I expect I'm going to need to build a new plugin as I'll have a lot of parsing as the messages tend to vary. But baby steps!

Thank you all in advance.

2

I'm making progress, I am able to pull part of the eventId:

#Syslog Input input { tcp { host => "192.168.1.180" port => 1514 } }

filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "MSSQL_DB_BACKUP" in [syslog5424_sd] {
grok {
match => {
"syslog5424_msg" => [ "(eventId=%{NUMBER:eventId}-" ]
}
}
}
}

output {
stdout { codec => rubydebug }
}

with result:

{ "message" => "<134>1 2016-11-09T17:06:56.858Z RVM154S014332 Rubrik - - [mdc@18060 ndc=\"MSSQL_DB_BACKUP_5282f305-967f-4d47-bd2f-a6a0a3f3ea0a:::526\" pid=\"10863\"] (eventId=1478711216846-2c5b4ce5-30fb-4b21-878f-6244d2b5c66d eventSeriesId=16a305d9-1f7a-4e80-81db-ad968beef4af objectId=b74f3482-5a49-4334-b7f8-2b47611cafdb@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed backup of Microsoft SQL Database 'MyTestDB' from 'ADP-MON01'", "@version" => "1", "@timestamp" => "2016-11-09T17:03:12.638Z", "host" => "192.168.1.102", "port" => 34825, "syslog5424_pri" => "134", "syslog5424_ver" => "1", "syslog5424_ts" => "2016-11-09T17:06:56.858Z", "syslog5424_host" => "RVM154S014332", "syslog5424_app" => "Rubrik", "syslog5424_sd" => "[mdc@18060 ndc=\"MSSQL_DB_BACKUP_5282f305-967f-4d47-bd2f-a6a0a3f3ea0a:::526\" pid=\"10863\"]", "syslog5424_msg" => "(eventId=1478711216846-2c5b4ce5-30fb-4b21-878f-6244d2b5c66d eventSeriesId=16a305d9-1f7a-4e80-81db-ad968beef4af objectId=b74f3482-5a49-4334-b7f8-2b47611cafdb@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed backup of Microsoft SQL Database 'MyTestDB' from 'ADP-MON01'", "eventId" => "1478711216846" }

But I'm still missing some fundamentals. I need all the pieces until the next white space to have the full ID and 'GREEDYDATA' gives me too much, 'DATA' gives me nothing, and trying to put in a custom pattern has create a lot of swearing and parse failures.

3

OK I've made some great progress. I'm capturing properly, I am confident in my parse patterns, but I'm not quite getting what I want in the output, here is my config:

Syslog Input input { tcp { host => "192.168.1.180" port => 1514 } }

filter {
grok {
match => {
"message" => '%{SYSLOG5424LINE}'
}
}
if "MSSQL_DB_BACKUP" in [syslog5424_sd] {
grok {
match => {
"syslog5424_msg" => [ "(eventId=%{DATA:eventId} ev" , "Database '%{DATA:dbname}'" , "from '%{DATA:bkuptarget}'" ]
}
}
}
}

output {
stdout { codec => rubydebug }
}

Here is the output:

{
"message" => "<134>1 2016-11-09T18:06:22.303Z RVM154S014332 Rubrik - - [mdc@18060 ndc="MSSQL_DB_BACKUP_e2560421-0d99-4966-bfee-b53f809cca4f_ea9696ef-e71c-4494-ae46-a8c5126e4b7e:::0" pid="10863"] (eventId=1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e eventSeriesId=1a896234-e5d6-44b6-bb50-687e11e531d8 objectId=87fcda14-e7fb-43cb-9428-75dbf9b7124c@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed manual backup of Microsoft SQL Database 'master' from 'ADP-MON01'",
"@version" => "1",
"@timestamp" => "2016-11-09T18:02:37.997Z",
"host" => "192.168.1.102",
"port" => 49122,
"syslog5424_pri" => "134",
"syslog5424_ver" => "1",
"syslog5424_ts" => "2016-11-09T18:06:22.303Z",
"syslog5424_host" => "RVM154S014332",
"syslog5424_app" => "Rubrik",
"syslog5424_sd" => "[mdc@18060 ndc="MSSQL_DB_BACKUP_e2560421-0d99-4966-bfee-b53f809cca4f_ea9696ef-e71c-4494-ae46-a8c5126e4b7e:::0" pid="10863"]",
"syslog5424_msg" => "(eventId=1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e eventSeriesId=1a896234-e5d6-44b6-bb50-687e11e531d8 objectId=87fcda14-e7fb-43cb-9428-75dbf9b7124c@7fc69c2b-a8c6-45b8-ac2e-4c0eabb36b8a status=Success) Completed manual backup of Microsoft SQL Database 'master' from 'ADP-MON01'",
"eventId" => "1478714782294-27c92103-7c7a-46aa-a704-87139f6b368e"
}

I'm now getting the full eventId, but my other pattersn aren't getting matched as I would expect. Thanks in advance.

4

That's because you're listing multiple patterns in the same grok filter and it'll stop after the first match. Use a single expression:

\(eventId=%{NOTSPACE:eventId} .* Database '(?<dbname>[^']+)' from '(?<bkuptarget>[^']+)'

}

5

Awesome. Does that mean that the functionality of:

match =>
"item" => [ "pattern1" , "pattern2" , "pattern3"]

is to find the FIRST matching pattern and then output? So if none of them match I'll get a parsefailure and if any one of them matches the first match will be recorded?

Really appreciate the help, has made a very big difference.

6

is to find the FIRST matching pattern and then output? So if none of them match I'll get a parsefailure and if any one of them matches the first match will be recorded?

Exactly.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.