Grok pattern for '|' character

Elastic Stack Logstash
1

I have 4 log files and a sample line from each of them is below. All these log files are under one index. How can i make one grok pattern that works for these two different format of log files?

The grok i have at the end only works for only the log starting with timestamp. How can I adjust my one grok pattern to extract - status. wrapper, timestamp and message from the "STATUS| wrapper|' type log?

2020-03-18 01:15:05,802 INFO  [qtp1694107926-3122] com.esq.rbac.sso.rest.SsoRest - loginSiteMinder; siteMinderRequestHeader=SM-USER; userName=null;
STATUS | wrapper  | 2019/12/03 05:18:27 | --> Wrapper Started as Service
2020-03-18 08:54:47,786 ERROR [qtp1763490167-117805] c.e.r.r.app.ExceptionMapperProvider - toResponse; exception={}
2020-03-05 21:39:24,898 INFO  [qtp1763490167-39667] c.e.d.contacts.rest.ObjectRoleRest - prepareJsonForContactsWithAvalableTime;dispatchContactAvalableTimeList=1
INFO   | jvm 832  | 2020/02/25 09:00:04 | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
%{DATESTAMP:datestamp}%{SPACE}%{LOGLEVEL:loglevel}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaclass}%{SPACE}%{GREEDYDATA:LOGMESSAGE}
2

Assuming you have used the grok debugger to check your patterns it demonstrates that the patterns have to match for them to parse. Unless you just want to throw GREEDYDATA at the end I don't see any other way of doing it, though I'm no expert.

My understanding is that the purpose of grok filters is to match the structure of your log files, therefore if two log files have completely different structures then one grok filter pattern will not do the job.

You could also use [log] [file] [path] to isolate which patterns are used for which log file.

3

@calanon, thanks. How will [log][file][path] work? The STATUS | wrapper file is called wrapper.log. And in logstash, i am already doing an if-else to separate indexes as below. So in logstash can I say-

filter {
   if [fields][tags] == "ob-webapi"{
      grok {
          match => { 
           "message" => ["%{DATESTAMP:timestamp}%{SPACE}%{GREEDYDATA}%{LOGLEVEL:loglevel}%{SPACE}%{NONNEGINT:anum}%{SPACE}%{JAVACLASS:javaClass}%{GREEDYDATA:logmessage} " ]
      }
    }
    }else if [fields][tags] == "rbac-logs"
    {
      match => {"log.file.path" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"}
     if [filename] == "wrapper" {
    grok {
	    match => {"message" => "%{WORD:level}" }
    }
}

So here first it matches tag from filebeat, then matches filename and then the grok. Would this work?

4

Figured a grok for removing the '|' character in the second log line above-

%{WORD:level} [|:]\s+%{GREEDYDATA:type}  [|:]\s+%{GREEDYDATA:timestamp} [|:]\s+%{GREEDYDATA}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.