Under one index I have multiple log files and one of those log file will use a different grok format than the other three log files. How can I apply grok pattern based on if statement on file name? For example,
- type: log
enabled: true
input_type: log
fields:
tags: ["obapp-dotnet"]
paths:
- 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt'
- 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt'
- 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt'
This is a block of my filebeat.yml. I want that under index "obapp-dotnet", log.tct and exceptionlog.txt use one grok pattern and dispatcher.log uses another type of grok pattern.
So,
if {dispatcher.log} use pattern1
else {exceptionlog.txt} use pattern 2
My filter block right now is-
filter {
if[fields][tags] =="obapp-java" {
grok {
break_on_match => false
match => {
"message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} ]
}
}
filter {
grok {
match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
}
}
} else if [fields][tags] == "obapp-dotnet" {
grok {
break_on_match => false
match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
if [path] = "dispatcher.log" {
grok{
match => {
"message" => [ ------- ]
}
}
else {
match => {
"message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
]
}
}
}
}
}
@Christian_Dahlqvist, what would you suggest here> Is the logstash in right format?
@mancharagopan Any suggestion for the logstash file? This format isnt working.
it should be if [path] == "dispatcher.log" {
Use Prospectors in filebeat configuration file to add fields.
Example:
filebeat:
prospectors:
- paths:
- 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt'
fields: {log_type: dispatcher}
Logstash:
filter {
if [fields][log_type] == "dispatcher" {
grok {
match => {
"message" => [ ------- ]
}
}
}
}
@mancharagopan Thanks. But by adding
the index will be dispatcher and not obapp-dotnet. How can we have obapp-dotnet as index and still a file identifier for each file?
@mancharagopan will this line be helpful in creating path as a field and then apply if statement on path since it will have file name such as dispatcher?
Although I tried this line and it doesnt create "path" as field. I would expect this line would create a field called Path: dispatcher.log. Or any other suggestion how I could have filename extracted in grok?
How are you separating indexes? Which field you are using?
can you share how the raw data looks like?
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
input_type: log
fields:
tags: ["obapp-dotnet"]
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt'
- 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt'
- 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\log.txt'
- 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\exceptionlog.txt'
- type: log
enabled: true
input_type: log
fields:
tags: ["obapp-java"]
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\*.log'
- 'C:\Program Files (x86)\ESQ SST\RBACService\logs\*.log'
- 'C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log'
this is block of filebeat.yml. The two tags get created as seperate indexes.
filter {
if[fields][log_type] =="obapp-java" {
grok {
#breaks if first match good, hence false to consider second match too.
break_on_match => false
match => {
"message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
]
}
}
filter {
grok {
match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
}
}
}
else if [fields][log_type] == "obapp-dotnet" {
grok {
break_on_match => false
match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
if [path] = "dispatcher.log" {
grok{
match => {
"message" => [ ]
}
}
else {
match => {
"message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
]
}
}
}
}
}
}
````onapp-dotnet````
'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt
27/01/2020 00:04:56 (null) INFO 10 OB.WebAPI.Business.Logic.OBWebAPIManager..ctor Entry Time - 27-01-2020 00:04:56.172
'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt' - \A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}
2020-01-02 15:19:40,710: (null) ERROR: (ESQ.CrossCutting.Instrumentation.ExceptionLoggingAspectAttribute.OnException(),Ln 0): OB.WebAPI.OBWebAPI.OB.WebAPI.Contracts.v1.IMS.GetIncidentArray failed. Message:Value cannot be null.
````onapp-java````
'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\dispatcher.log
2020-01-27 03:11:21,038 [DispatcherScheduler_Worker-2] INFO o.a.c.h.HttpMethodDirector - Retrying request
'C:\Program Files (x86)\ESQ SST\RBACService\logs\rbac.log
2020-01-27 00:21:23,337 INFO [qtp2141445292-70219] org.eclipse.jetty.server.session - Session node0mppnjr4chzg1eps3zohnru4217853 already being invalidated
C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log
20191203 04:50:54.671 [main] INFO o.s.b.f.x.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [config/obapi-custom-data-config.xml]
Under first index are two file paths mentioned with a sample of log type stored in them. So with index onapp-dotnet, file path is "c:....log.txt' and the format of log in it is in next line.