Help with Grok Pattern for SAS log

Elastic Stack Logstash
1

Hi!

I have just started playing with the Elastic Stack and I am a complete noob, so please treat this as such.

I need help in constructing a grok filter for some logs i am parsing to Logstash, more specifically I need help breaking down the "GREEDYDATA" portion so i can use the information within.

Here are a few log file lines for reference:

2017-02-28T12:33:27,569 INFO  [00000035] :a196046 - NOTE: DATA statement used (Total process time):
2017-02-28T12:33:27,570 INFO  [00000035] :a196046 -       real time           0.55 seconds
2017-02-28T12:33:27,570 INFO  [00000035] :a196046 -       user cpu time       0.02 seconds
2017-02-28T12:33:27,570 INFO  [00000035] :a196046 -       system cpu time     0.18 seconds

I am using the following Grok filter:

filter {
 if [type] == "saslog" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{GREEDYDATA:msg}" }
      }
    }
 }

How can I break this down further so that I can use the information in the GREEDYDATA such as CPU time?

2

https://grokdebug.herokuapp.com/ and https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns are your friends when creating Grok patterns. I'm not sure how you want to parse the log, so I just give an example

To parse this line

2017-02-28T12:33:27,570 INFO  [00000035] :a196046 -       user cpu time       0.02 seconds

use

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} user cpu time %{SPACE} %{NUMBER:user_cpu_time} seconds

so the config could look like:

filter {
 if [type] == "saslog" {
    grok {
        match => { "message" => [
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} user cpu time %{SPACE} %{NUMBER:user_cpu_time} seconds",
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} system cpu time %{SPACE} %{NUMBER:system_cpu_time} seconds"
                    
                ]
            }
        }
    }
 }

Add more Grok patterns as necessary.

1 Like
3

Thank you Anh!
It worked like a charm.

For anyone else coming across this, my grok filter ended up looking like this:

filter {
 if [type] == "saslog" {
    grok {
        match => { "message" => [
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} real time %{SPACE} %{NUMBER:real_time} seconds",
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} user cpu time %{SPACE} %{NUMBER:user_cpu_time} seconds",
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} system cpu time %{SPACE} %{NUMBER:system_cpu_time} seconds",
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} memory %{SPACE} %{NUMBER:memory} seconds",
                    "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{SPACE} OS Memory %{SPACE} %{NUMBER:OS_memory} seconds"
                    
                ]
            }
        }
    }
        if "_grokparsefailure" in [tags] {      
        drop { }
 }
}

Just added the code to drop any logs that do not match.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.