Apply grok pattern based on logfile name

Elastic Stack Logstash
1

Under one index I have multiple log files and one of those log file will use a different grok format than the other three log files. How can I apply grok pattern based on if statement on file name? For example,

- type: log

  enabled: true
  input_type: log
  fields:
    tags: ["obapp-dotnet"]

  paths:
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt'

This is a block of my filebeat.yml. I want that under index "obapp-dotnet", log.tct and exceptionlog.txt use one grok pattern and dispatcher.log uses another type of grok pattern.
So,

if {dispatcher.log} use pattern1
else {exceptionlog.txt} use pattern 2

My filter block right now is-

filter {
  if[fields][tags] =="obapp-java" {
    grok {
      break_on_match => false
      match => {
        "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} ]
      }
    }
filter {
    grok {
       match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
    }
  }
} else if [fields][tags] == "obapp-dotnet" {
    grok {
      break_on_match => false
      match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
      if [path] = "dispatcher.log" {
	     grok{
		    match => {
               "message" => [ ------- ]
	          }
        	 }
      else {
	  match => {
        "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
          ]
      }
	 }
    }
}
}
2

@Christian_Dahlqvist, what would you suggest here> Is the logstash in right format?

3

@mancharagopan Any suggestion for the logstash file? This format isnt working.

4

it should be if [path] == "dispatcher.log" {

5

Use Prospectors in filebeat configuration file to add fields.

Example:

filebeat:
  prospectors:
    - paths:
        - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt'
      fields:  {log_type: dispatcher}

Logstash:
filter {
  if [fields][log_type] == "dispatcher" {
    grok {
	match => {
              "message" => [ ------- ]
        }
    }
  }
}
6

@mancharagopan Thanks. But by adding

the index will be dispatcher and not obapp-dotnet. How can we have obapp-dotnet as index and still a file identifier for each file?

7

@mancharagopan will this line be helpful in creating path as a field and then apply if statement on path since it will have file name such as dispatcher?

Although I tried this line and it doesnt create "path" as field. I would expect this line would create a field called Path: dispatcher.log. Or any other suggestion how I could have filename extracted in grok?

8

How are you separating indexes? Which field you are using?

9

can you share how the raw data looks like?

10 
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true
  input_type: log
  fields:
    tags: ["obapp-dotnet"]

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\log.txt'
    - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\exceptionlog.txt'    
- type: log

  enabled: true
  input_type: log
  fields:
    tags: ["obapp-java"]

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - 'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\*.log'
    - 'C:\Program Files (x86)\ESQ SST\RBACService\logs\*.log'
    - 'C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log'

this is block of filebeat.yml. The two tags get created as seperate indexes.

filter {
  if[fields][log_type] =="obapp-java" {
    grok {
	#breaks if first match good, hence false to consider second match too.
      break_on_match => false
      match => {
        "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
         ]
      }
    }
filter {
    grok {
       match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
    }
  }
} 

else if [fields][log_type] == "obapp-dotnet" {
    grok {
      break_on_match => false
	  match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
	  if [path] = "dispatcher.log" {
	     grok{
		    match => {
               "message" => [  ]
			   }
			 }
      else {
	  match => {
        "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
          ]
      }
	 } 
    }
}
}
}
11 
````onapp-dotnet````
'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt
27/01/2020 00:04:56   (null)                  INFO   10   OB.WebAPI.Business.Logic.OBWebAPIManager..ctor Entry Time - 27-01-2020 00:04:56.172 
'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt' - \A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
2020-01-02 15:19:40,710:               (null) ERROR: (ESQ.CrossCutting.Instrumentation.ExceptionLoggingAspectAttribute.OnException(),Ln 0): OB.WebAPI.OBWebAPI.OB.WebAPI.Contracts.v1.IMS.GetIncidentArray failed. Message:Value cannot be null.
````onapp-java````
'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\dispatcher.log 
2020-01-27 03:11:21,038 [DispatcherScheduler_Worker-2] INFO   o.a.c.h.HttpMethodDirector - Retrying request
'C:\Program Files (x86)\ESQ SST\RBACService\logs\rbac.log 
2020-01-27 00:21:23,337 INFO  [qtp2141445292-70219] org.eclipse.jetty.server.session - Session node0mppnjr4chzg1eps3zohnru4217853 already being invalidated
C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log
20191203 04:50:54.671 [main] INFO  o.s.b.f.x.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [config/obapi-custom-data-config.xml]

Under first index are two file paths mentioned with a sample of log type stored in them. So with index onapp-dotnet, file path is "c:....log.txt' and the format of log in it is in next line.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.