Can't remove_field - help please

Elastic Stack Logstash
1

I am very new with Logstash, trying to create my first config. I am using 1.5.6 version.
As I am testing, I wanted to try to use db2 log file as input. Got the multiline thing working fine. Example of my event:
"2016-02-26-14.51.45.577696-360 I959779E406 LEVEL: Warning\nPID : 24767 TID : 140656464664352 PROC : db2star2\nINSTANCE: db2inst1 NODE : 000\nHOSTNAME: opsmgmt01.wdc01.unica.com\nFUNCTION: DB2 UDB, base sys utilities, sqleReleaseStStLockFile, probe:14016\nMESSAGE : Released lock on the file:\nDATA #1 : String, 39 bytes\n/home/db2inst1/sqllib/ctrl/db2stst.0000"

I want to get rid of everything after LEVEL:Warning.
This is my filter:
filter {
grok {
match => { message => "%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND}[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" }
}
mutate { remove_field => [ "therest" ] }
}

I used grok debugger and it displays "therest" correctly.
My output:
output {
if [loglevel] not in ["Info","Event"] {
stdout { codec => rubydebug }
email { to => "mivkovic@xyz.com"
subject => "check db2 instance log"
from => "root@%{host}"
body => "Here is the event: %{message}" }
}
}

On my stdout, I see message as the whole thing, including "therest" field. Same thing for my email message body.
How does my new message becomes everything but "therest" field?
Many thanks!

2

Does your pattern actually match 100%?

3

I think you may be misunderstanding exactly what the grok filter does. It creates fields from the %{message} based on the pattern provided, but does not alter the original %{message} which will always be what has been received from the input filter, in this case your log entry.

Your code groks the message to create fields, but your output filter is sending you the original %{message} which will always contain %{therest}.

What you need to do is create a new message from the grok fields and send this instead.

Try something like this:

match => { message => "(?<runtime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}-%{HOUR}.%{MINUTE}.%{SECOND}[-]%{INT})%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" }

mutate { add_field => { "new_message" => "%{runtime}-%{loglevel}"} }

output { 
if [loglevel] not in ["Info","Event"] {
stdout { codec => rubydebug }
email { to => "mivkovic@xyz.com"
subject => "check db2 instance log" 
from => "root@%{host}"
body => "Here is the event: %{new_message}" }
}
}
4

Yes it does. Checked it in grok debugger.

5

Thanks, I was suspecting my "message" always remains the same, in output. What still doesn't make sense to me is that, when I remove_field in grok, shouldn't stdout { codec => rubydebug } not display it? I understand remove_field like this: takes input and removes the field so it is not shown in output. If not, what does it remove it from then?

About your (?>runtime:...), that doesn't work. I assume you wanted to make it so my whole date/time thing is returned as a runtime field. I have tried to play with various combinations there but none worked. I would think this should work (simplified with just year and month):
match => { message => "%{(%{YEAR}[-]%{MONTHNUM}):runtime}
but it doesn't. I followed this:
%{SYNTAX:SEMANTIC}
so my semantic is runtime. And my syntax is in (), combination of several YEAR and MONTH.

Many thanks!

6

Sorry I had a typo on the pattern, which I have corrected in the original post above, so you should now get a field called runtime with a value of:

2016-02-26-14.51.45.577696-360

Could you please post an example of the exact output you are getting from the ruby debug.

7

Thanks, it works now! This must be the Oniguruma syntax "(?<field_name>the pattern here)", right? I had tried that too, but didn't realize I need to have <>. :slightly_smiling:

This is my filter:
filter {
grok {
match => { message => "(?%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND})[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" }
}
mutate { remove_field => [ "therest" ] }
mutate { add_field => {"new_message" => "%{runtime}-%{loglevel}"} }
}

And this is my output:

Logstash startup completed
{
"@timestamp" => "2016-02-29T18:44:29.498Z",
"message" => "2016-02-29-12.44.28.741765-360 E988348E380 LEVEL: Warning (OS)\nPID : 17367 TID : 140140202448672 PROC : db2star2\nINSTANCE: db2inst1 NODE : 000\nHOSTNAME: opsmgmt01.wdc01.tealeaf.ibmcloud.com\nFUNCTION: DB2 UDB, SQO Memory Management, sqloMemCreateSingleSegment, probe:100\nCALLED : OS, -, shmget OSERR: EEXIST (17)",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"host" => "opsmgmt01.xyz.com",
"path" => "/home/db2inst1/sqllib/db2dump/db2diag.log",
"type" => "db2-log",
"runtime" => "2016-02-29-12.44.28.741765",
"loglevel" => "Warning",
"new_message" => "2016-02-29-12.44.28.741765-Warning"
}

I see that all fields are actually returned properly. How would I use remove_field here, instead of add_field?
So if I wanted to output everything but 1 field, I would think remove_field would be perfect for it, instead of creating new field to include all but 1 I don't want.
Many thanks!
Marjana

8

I am getting a bit confused as to exactly what your requirement is. Your original post had a requirement to generate an email with part of the message as the body of the email. If this is all you need, then what you have should deliver on that requirement.

So if you just need to strip off the end of the message, then you could change the example I posted slightly and complete the process using just the grok filter, but this will only create one field:

Example 1:

 filter {
        grok {
            match => { message => "(?<new_message>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}-%{HOUR}.%{MINUTE}.%{SECOND}[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel})"}
        }
        mutate { remove_field => [ "message", "loglevel"] }
 }

This will deliver the following output:

{
     "@version" => "1",
     "@timestamp" => "2016-03-01T09:48:41.659Z",
     "host" => "server1",
     "new_message" => "2016-02-26-14.51.45.577696-360 I959779E406 LEVEL: Warning"
 }

This new_message field can then be used in the body of your email.

If your requirement is to break the message up into individual fields then you already have what you need:

Example 2:

filter {
      grok {
           match => { message => "(?<runtime>%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND})[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" }
      }
            mutate { remove_field => ["message"]}
 }

Delivers the following output:

 {
     "@version" => "1",
     "@timestamp" => "2016-03-01T09:54:36.491Z",
     "host" => "server1",
     "runtime" => "2016-02-26-14.51.45.577696",
     "loglevel" => "Warning",
     "therest" => "\\nPID : 24767 TID : 140656464664352 PROC : db2star2\\nINSTANCE: db2inst1 NODE : 000\\nHOSTNAME: opsmgmt01.wdc01.unica.com\\nFUNCTION: DB2 UDB, base sys utilities, sqleReleaseStStLockFile, probe:14016\\nMESSAGE : Released lock on the file:\\nDATA #1 : String, 39 bytes\\n/home/db2inst1/sqllib/ctrl/db2stst.0000"
 }

So now you can do what you like with these fields and if you need to break up "therest" then just work on your grok filter to split this up further.

9

Thanks, this helps a lot!!All set now.