Try to parse different log formats in one file?

Elastic Stack Logstash
1

Hi, I have situation where two different formats end up in the same log file.

I have the standard log4j format and that parses fine. But I also get google glog format.

The glog looks like:

I0807 19:07:40.477815 13465 checker_process.cpp:972] HTTP health check for task 'auth-api-stg-raange.c1681d01-95c2-11e8-818f-1a706db5e5af' returned: 200

The log4j pattern works as the message does not get tagged as _grokparsefailure.

This is the filter I'm using

grok {
patterns_dir => "patterns/"
match => {
"message" =>
[
"%{TIMESTAMP_ISO8601:log_timestamp} [%{NOTSPACE:thread}] %{LOGLEVEL:log_level} * (?[A-Za-z0-9$_.]+) - %{GREEDYDATA:log_message}",
"%{LOGLEVEL:log_level}%{TIMESTAMP_GLOG:log_timestamp} %{GREEDYDATA:log_message}"
]
}
}

GLOG pattern is defined as:

TIMESTAMP_GLOG %{MONTHNUM}%{MONTHDAY} %{TIME}

2

I came across this situation in one of my implementation , i wrote two grok's in filter section something like beloe , please ignore the syntax

    >     filter {
    > 
    >     grok { 
             grok pattern for first log
    >     }
    >     if [_grokparsefailure] in [tags] {
    >     grok {
           grok pattern for second type of log
    >     }
    >     if [GreedyData field] {
    >     remove "grokeparsefailure" from tags
    >      }
    >     }
1 Like
3

The grok expression you have for the glog file doesn't match that actual log because LOGLEVEL doesn't match "I". Try e.g. (?<log_level>\w)%{TIMESTAMP_GLOG}... instead.

4

Ok I will try when I have a bit of time. Thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.