# Issue with Index filtering within Logstash

**URL:** <https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150>\
**Category:** Logstash\
**Created:** [May 25, 2017, 4:09pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150 "2017-05-25T16:09:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [May 25, 2017, 4:09pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150/1 "2017-05-25T16:09:21Z")

</div>

Hello, we are currently using 5.3 of Elastic stack. We have two different teams using logstash so we have two different logstash conf files. One file is using UDP input with port 4558 and the other is using beats input with port 7777. Both outputs are sending to ES and have completely two different indexes. It seems the one with the beats input data is getting sent using the other index also. So we have duplicate data, data under the wrong index and errors in the logs for logstash because it is trying to parse the data as a JSON but there is no json data active. Is there something that can be causing this?

Thanks,  
Kenneth

---

<div class="post-metadata">

**Author:** ![orhiee](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@orhiee](https://discuss.elastic.co/u/orhiee)\
**Post date:** [May 25, 2017, 4:14pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150/2 "2017-05-25T16:14:18Z")

</div>

logstash filters and outputs are applied to all the data that comes in so if you have an event coming in "event1" this event will go through all the filters and the outputs.

you can add tags|types at the input level and put IFs ont he begining of your output to filter where its applied

hope this helps 🙂

what i have is something like this:

```
input {
  beats {
    port => XXX
  }
}

output {
 if [type] == "wineventlog" {
  elasticsearch {
    hosts => "XXXXXX:9200"
    manage_template => false
    index => "XXXXXX-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
 }
}
```

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [May 25, 2017, 4:20pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150/3 "2017-05-25T16:20:10Z")

</div>

I do have that setup which is weird to me.

output {  
if [type] == "app-logs" {  
elasticsearch {  
hosts =\> "es-hostname:9200"  
manage\_template =\> false  
index =\> "app-filebeat-%{+YYYY.MM.dd}"  
}  
}

the UDP logstash file does not though as I am not sure if there is a way to set a document type for that?

---

<div class="post-metadata">

**Author:** ![orhiee](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@orhiee](https://discuss.elastic.co/u/orhiee)\
**Post date:** [May 25, 2017, 7:14pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150/4 "2017-05-25T19:14:19Z")

</div>

so you only have beats data in the beats index ? because you are filtering it but how is your other output ?

the config you pasted will make sure only "app-logs" write to "app-filebeat-\*" bu it wont stop "app-logs" from writing to other index 😉

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2017, 7:14pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150/5 "2017-06-22T19:14:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.