Logstash - extraction substring from the raw message

Elastic Stack Logstash
1

Hi Everyone,

I'm trying to see if there is any logstash plugin which we can use to perform a quick filtering on 2 substrings without need to parse the entire message using grok command:

An example of the log is below and I'm only interesting to extract the 2 substrings in Bold.

<12> May 1 11:46:42 EMCstorevntd: [fmt=evt] [evtid=1072] [date=2018-05-01T15:43:35Z] [symid=000196801796] [Device=00037] [sev=warning] = SRDF R2 device not ready."

With Logstatch, can I do similar regular expression like (symid=)[0-9]{12} to extract the symmetrix ID number and (sev=)(warning|critical) to extract the severity ?

thanks by advanced to share feedback

2

grok doesn't need to match against the whole pattern; it can be used to extract specific bits, and the break_on_match => false directive allows us to extract multiple bits independent of their order:

filter {
  grok {
    break_on_match => false
    match => {
      "message" => ["\[symid=%{NUMBER:symid}\]", "\[sev=%{LOGLEVEL:sev}\]"]
    }
  }
}

When using break_on_match => false, you'll want to make sure your the patterns begin with as specific a string as possible, which will empower the matcher to avoid unnecessary work.

3

Thanks you very much. It works perfectly.
Can I put in the same grok filter break_on_match =>fais and break_on_match =true to concatenate multiple filter matching ?
Something like that :

filter {
grok {
break_on_match => true
match => {
"message" => [xxxxx]
break_on_match => false
match => {
"message" => [xxxxx]
}
}

Thanks again to share your knowledge.

4

no; the break_on_match directive applies to the entire grok filter instance and cannot be flip/flopped.

You can, however, use multiple grok filters:

filter {
  # without the `break_on_match` directive (or when explicitly set to `true`),
  # once a match is found, the remaining patterns are not run
  grok {
    match => {
      "message" => [xxxxx, yyyy]
    }
  }

  # when `break_on_match` is set to `false`, grok will attempt to capture using
  # all patterns, even after it finds a match.
  grok {
    break_on_match => false
    match => {
      "message" => [zzzzz, wwww]
    }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.