# How to parse different format logs from same directory

**URL:** <https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772>\
**Category:** Logstash\
**Created:** [July 5, 2016, 6:39pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772 "2016-07-05T18:39:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 5, 2016, 6:39pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772/1 "2016-07-05T18:39:39Z")

</div>

I have multiple log files in the same directory, that I want to run through Logstash, every file is having different format than other.  
My log files can be named randomly by different network providers (I dont have control on how do they name log files). I want to parse based on certain formats they have.

**For example:** under logs directory there are log files with different format  
$ /path/to/logs

> abc.log  
> beats.log  
> access.log

However, i was able to parse each log file separately by launching logstash instance to the specific format of log file (While parsing access.log file i'm using access.conf file which has the matching grok filter to parse the access.log data format).

Should I run as many instances as I have different types of logs?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 5, 2016, 6:47pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772/2 "2016-07-05T18:47:31Z")

</div>

With the grok filter you can list multiple expressions and have the filter try them in order until one matches. Another option could be to use conditionals to classify the events based on what they look like, e.g. like this:

```nohighlight
filter {
  if [message] =~ /some regexp that matches one type of event/ {
    mutate {
      replace => {
        "type" => "some type"
      }
    }
  }
}

```

Then use additional conditional blocks based on the `type` field.

---

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 20, 2016, 2:19pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772/3 "2016-07-20T14:19:40Z")

</div>

Magnus, thank you for response. I'm using Filebeat to ship logs to Logstsash. Can we set some type on the Filebeat, to use conditional filters in Logstash.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 20, 2016, 2:33pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772/4 "2016-07-20T14:33:35Z")

</div>

Set the prospector's [`document_type` option](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_document_type).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:47am UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772/5 "2017-07-06T04:47:17Z")

</div>