# Apply grok pattern based on logfile name **URL:** \ **Category:** Logstash\ **Created:** [January 28, 2020, 6:42pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901 "2020-01-28T18:42:13Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 28, 2020, 6:42pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/1 "2020-01-28T18:42:13Z")
Under one index I have multiple log files and one of those log file will use a different grok format than the other three log files. How can I apply grok pattern based on if statement on file name? For example, ```auto - type: log enabled: true input_type: log fields: tags: ["obapp-dotnet"] paths: - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt' - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt' - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt' ``` This is a block of my filebeat.yml. I want that under index "obapp-dotnet", log.tct and exceptionlog.txt use one grok pattern and dispatcher.log uses another type of grok pattern. So, ```auto if {dispatcher.log} use pattern1 else {exceptionlog.txt} use pattern 2 ``` My filter block right now is- ```auto filter { if[fields][tags] =="obapp-java" { grok { break_on_match => false match => { "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass}] } } filter { grok { match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"] } } } else if [fields][tags] == "obapp-dotnet" { grok { break_on_match => false match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"] if [path] = "dispatcher.log" { grok{ match => { "message" => [-------] } } else { match => { "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} ] } } } } } ``` ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 28, 2020, 7:07pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/2 "2020-01-28T19:07:23Z")
@Christian_Dahlqvist, what would you suggest here\> Is the logstash in right format? ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 28, 2020, 11:56pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/3 "2020-01-28T23:56:58Z")
@mancharagopan Any suggestion for the logstash file? This format isnt working. ---
**Author:** ![mancharagopan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mancharagopan/32/60266_2.png) [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Post date:** [January 29, 2020, 3:02am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/4 "2020-01-29T03:02:40Z")
> [@Mehak\_Bhargava](#): > > if [path] = "dispatcher.log" { it should be `if [path] == "dispatcher.log" {` ---
**Author:** ![mancharagopan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mancharagopan/32/60266_2.png) [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Post date:** [January 29, 2020, 3:10am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/5 "2020-01-29T03:10:39Z")
Use Prospectors in filebeat configuration file to add fields. Example: ```auto filebeat: prospectors: - paths: - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\dispatcher.txt' fields: {log_type: dispatcher} ``` ```auto Logstash: filter { if [fields][log_type] == "dispatcher" { grok { match => { "message" => [-------] } } } } ``` ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 29, 2020, 7:20am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/6 "2020-01-29T07:20:58Z")
@mancharagopan Thanks. But by adding > [@mancharagopan](#): > > fields: {log\_type: dispatcher} the index will be dispatcher and not obapp-dotnet. How can we have obapp-dotnet as index and still a file identifier for each file? ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 29, 2020, 7:23am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/7 "2020-01-29T07:23:18Z")
> [@Mehak\_Bhargava](#): > > match =\> ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}.log"] @mancharagopan will this line be helpful in creating path as a field and then apply if statement on path since it will have file name such as dispatcher? Although I tried this line and it doesnt create "path" as field. I would expect this line would create a field called Path: dispatcher.log. Or any other suggestion how I could have filename extracted in grok? ---
**Author:** ![mancharagopan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mancharagopan/32/60266_2.png) [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Post date:** [January 29, 2020, 8:28am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/8 "2020-01-29T08:28:46Z")
How are you separating indexes? Which field you are using? ---
**Author:** ![mancharagopan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mancharagopan/32/60266_2.png) [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Post date:** [January 29, 2020, 8:30am UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/9 "2020-01-29T08:30:02Z")
can you share how the raw data looks like? ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 29, 2020, 4:19pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/10 "2020-01-29T16:19:09Z")
```auto filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true input_type: log fields: tags: ["obapp-dotnet"] # Paths that should be crawled and fetched. Glob based paths. paths: - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt' - 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt' - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\log.txt' - 'C:\Program Files (x86)\ESQ SST\Logs\IMSService\exceptionlog.txt' - type: log enabled: true input_type: log fields: tags: ["obapp-java"] # Paths that should be crawled and fetched. Glob based paths. paths: - 'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\*.log' - 'C:\Program Files (x86)\ESQ SST\RBACService\logs\*.log' - 'C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log' ``` this is block of filebeat.yml. The two tags get created as seperate indexes. ```auto filter { if[fields][log_type] =="obapp-java" { grok { #breaks if first match good, hence false to consider second match too. break_on_match => false match => { "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} ] } } filter { grok { match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"] } } } else if [fields][log_type] == "obapp-dotnet" { grok { break_on_match => false match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"] if [path] = "dispatcher.log" { grok{ match => { "message" => [] } } else { match => { "message" => [\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} ] } } } } } } ``` ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [January 29, 2020, 4:21pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/11 "2020-01-29T16:21:59Z")
`````auto ````onapp-dotnet```` 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\log.txt 27/01/2020 00:04:56 (null) INFO 10 OB.WebAPI.Business.Logic.OBWebAPIManager..ctor Entry Time - 27-01-2020 00:04:56.172 'C:\Program Files (x86)\ESQ SST\Logs\OBWebAPI\exceptionlog.txt' - \A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA}%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA}%{SPACE}%{JAVACLASS:javaClass} 2020-01-02 15:19:40,710: (null) ERROR: (ESQ.CrossCutting.Instrumentation.ExceptionLoggingAspectAttribute.OnException(),Ln 0): OB.WebAPI.OBWebAPI.OB.WebAPI.Contracts.v1.IMS.GetIncidentArray failed. Message:Value cannot be null. ````onapp-java```` 'C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\dispatcher.log 2020-01-27 03:11:21,038 [DispatcherScheduler_Worker-2] INFO o.a.c.h.HttpMethodDirector - Retrying request 'C:\Program Files (x86)\ESQ SST\RBACService\logs\rbac.log 2020-01-27 00:21:23,337 INFO [qtp2141445292-70219] org.eclipse.jetty.server.session - Session node0mppnjr4chzg1eps3zohnru4217853 already being invalidated C:\Program Files (x86)\ESQ SST\OBAPI\logs\*.log 20191203 04:50:54.671 [main] INFO o.s.b.f.x.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [config/obapi-custom-data-config.xml] ````` Under first index are two file paths mentioned with a sample of log type stored in them. So with index onapp-dotnet, file path is "c:....log.txt' and the format of log in it is in next line. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2020, 4:25pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-logfile-name/216901/12 "2020-02-26T16:25:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.