# How to parse different types log files in logstash?

**URL:** https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462
**Category:** Logstash
**Created:** [September 14, 2016, 7:03am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462 "2016-09-14T07:03:47Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![dvkawa](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dvkawa](https://discuss.elastic.co/u/dvkawa)
#### Post date: [September 14, 2016, 7:03am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/1 "2016-09-14T07:03:47Z")

</div>

```
Hi,

I have two different weblogic log files, my conf file looks like below but it doesn't work:
                input {
                  file {
                        path => "D:/<dir>/logs/ecomm_vws02_app"
                        start_position => beginning 
                        ignore_older => 0 
                	sincedb_path => "/dev/null"
                	type => "application"
                    }
                     file {
                        path => "D:/<dir>/logs/ecomm_vws02_access.201609121200"
                        start_position => beginning 
                        ignore_older => 0 
                	sincedb_path => "/dev/null"
                	type => "access"
                    }
                }
            filter	{  

                          if [type] == "application" {
            		grok {
            		        patterns_dir => "D:/<dir>/logstash-all-plugins-2.4.0/logstash-2.4.0/patterns"
            			match => ["message","%{WEBLOGIC_POC}|%{WEBLOGIC_POC1}"]
            		}
            		
            		date {
            			match => ["timestamp","EEE MMM dd HH:mm:ss z yyyy"]
            		}
            	}
            	 if [type] == "access" {
            		grok {
            		       			match => { "message" => "%{COMBINEDAPACHELOG}"}
            		}
            		
            		date {
            			match => ["timestamp","EEE MMM dd HH:mm:ss z yyyy"]
            		}
            	}
            }
            output {
               if [type] == "application" {
                elasticsearch {
                    hosts => ["localhost:9200"]
            	index => "prod_logs_data-%{+YYYY.MM.dd}"
            	}
                 }
            	 if [type] == "access"{
            	 elasticsearch {
                     hosts => ["localhost:9200"]
                     index => "prod_accesslogs_data-%{+YYYY.MM.dd}"
            	 }
                }
            }`Preformatted text`
```

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 14, 2016, 7:05am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/2 "2016-09-14T07:05:04Z")

</div>

Please format your code with the `</>` button, it’s very hard to read as it is.

---

<div class="post-metadata">

### Author: ![dvkawa](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dvkawa](https://discuss.elastic.co/u/dvkawa)
#### Post date: [September 14, 2016, 7:09am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/3 "2016-09-14T07:09:11Z")

</div>

Does it look better?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 14, 2016, 8:54am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/4 "2016-09-14T08:54:38Z")

</div>

What isn't working?

---

<div class="post-metadata">

### Author: ![dvkawa](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dvkawa](https://discuss.elastic.co/u/dvkawa)
#### Post date: [September 14, 2016, 8:59am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/5 "2016-09-14T08:59:45Z")

</div>

Hi, Looks like its not able to understand the two file inputs, its reading the second file - ecomm\_vws02\_access.201609121200 alone and creating only-prod\_logs\_data-%{+YYYY.MM.dd} index, with gorkparse failure and not creating the second index -prod\_accesslogs\_data-\*

Please advice, thanks

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 14, 2016, 9:27am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/6 "2016-09-14T09:27:22Z")

</div>

Is it reading the other file at all? I suggest you disable the (somewhat) working file input and focus debugging on what doesn't work at all and then fix one thing at a time. This will be easier to debug if you replace the elasticsearch outputs with a single `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

### Author: ![dvkawa](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dvkawa](https://discuss.elastic.co/u/dvkawa)
#### Post date: [September 14, 2016, 12:22pm UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/7 "2016-09-14T12:22:29Z")

</div>

Thanks Magnus - But the issue is, if I specify a single input location, it works perfectly well. But If i place both types of logs in the input directory, it reads the FIRST one ONLY. Second file isnt getting read at all.Anything wrong in my syntax? Please suggest, its little critical. Many Thanks.`Preformatted text`

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 14, 2016, 12:44pm UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/8 "2016-09-14T12:44:36Z")

</div>

I haven't heard of such behavior before. I'd kick the log level to max with `--debug` and use other debugging tools (lsof, strace, ...) if needed. Is it opening both files? Is it reading from both files? Et cetera.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:38am UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462/9 "2017-07-06T04:38:31Z")

</div>


