# Best practice for log filter for multiple log

**URL:** <https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860>\
**Category:** Logstash\
**Created:** [May 15, 2018, 7:06am UTC](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860 "2018-05-15T07:06:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\
**Post date:** [May 15, 2018, 7:06am UTC](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860/1 "2018-05-15T07:06:15Z")

</div>

HI,  
I am working on setting up ELK stack for aggregating my IIS and IVR logs. Both services run on different servers. I have already set up logstash pipeline for IIS logs. i need to set it up for IVR logs. Shall i use the same pipeline and add the filter for IVR logs? or create a new pipeline with new index?

I use file beat to ship the logs and using fields prospectors to identify the IIS and ivr logs.

> fields: {log\_type: iis}

Thanks.

---

<div class="post-metadata">

**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\
**Post date:** [May 15, 2018, 9:31pm UTC](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860/2 "2018-05-15T21:31:55Z")

</div>

How you do your pipeline is up to you. This is how I have mine set up:

In /logstash/conf.d directory, I have "01-input.conf" as my file that handles all of my inputs.

Then, I have multiple filter.conf files numbered 10-20 like 11-filter-iislogs.conf and 12-filter-winevens.conf. Each of those files handles only those types of logs.

Lastly, I have 99-output.conf that handles all the output options like:

```
output {
    if [type] == "iislogs" {
      elasticsearch {
        hosts => ["localhost:9200"]
        index => "iislogs-%{+YYYY-MM-dd}"
      }
    }
    if [type] == "winevents" {
      elasticsearch {
        hosts => ["localhost:9200"]
        index => "winevents-%{+YYYY-MM-dd}"
      }
    }
}

```

and so on. For me, this makes it easier to modify a particular section rather than having one huge file to deal with.

I personally would recommend creating an index for each type of data you are using because by default there is a limit to the number of fields you can have per index, which depending on your version of Elastic may be 1000. You can change this default value, but I find the less I have to do manually, the better.

Having separate indices also improves on Kibana/Elasticsearch performance when creating visualizations because it will only be looking in the specified index for data and not everything you have.

I hope this helps.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2018, 9:33pm UTC](https://discuss.elastic.co/t/best-practice-for-log-filter-for-multiple-log/131860/3 "2018-06-12T21:33:54Z")

</div>

Closed on June 12, 2018, 9:33pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.