# Logstash pipeline output | duplicate messages ending up indexes
**URL:** https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878
**Category:** Logstash
**Created:** [September 17, 2019, 7:37pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878 "2019-09-17T19:37:30Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@deeps](https://discuss.elastic.co/u/deeps)
#### Post date: [September 17, 2019, 7:37pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/1 "2019-09-17T19:37:30Z")
In the /etc/logstash/conf.d/ directory I have configured 2 pipeline files to read messages from kafka topic/s and send it to data nodes in the cluster. (1.conf and 2.conf)
```
[root@ingest1 conf.d]# egrep -w "topics|index" * | uniq
1.conf: topics => ["events"]
1.conf: index => "events-%{+YYYY.MM.dd}"
2.conf: topics => ["input"]
2.conf: index => "input-%{+YYYY.MM.dd}"
```
But if I produce a message to "events" topic, the message is ending up in 2 indexes. Same with other topic also.
Am I missing anything?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 17, 2019, 7:40pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/2 "2019-09-17T19:40:47Z")
All files in the config directory are concatenation into a smaller neglected pipeline, meaning that data from all inputs go through all filters and are sent to all outputs unless you use conditionals to control the glow.
---
### Author:  [@deeps](https://discuss.elastic.co/u/deeps)
#### Post date: [September 17, 2019, 7:43pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/3 "2019-09-17T19:43:36Z")
@Christian_Dahlqvist thanks for the quick response.
could you also guide me how to solve this?
_ex: i want 1.conf to read "events" topic, apply pipeline filter **and just output to its index.** _
TIA!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 17, 2019, 11:13pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/4 "2019-09-17T23:13:51Z")
If the two configurations are completely separate from input to output I would strongly suggest using [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html). If there is overlap, or you are stuck on an old version then you can use something like
```
add_field => { inputTopic => "events" }
```
(with two different value for inputTopic) on the inputs to distinguish them, then use
```
output {
if [inputTopic] == "events" {
elasticsearch {
...
}
}
}
```
to send them to different end-points.
Even better, since you are using a kafka input, you can have the input decorate the [metadata](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#_metadata_fields) with the topic name and then make the output configuration conditional upon that.
---
### Author:  [@deeps](https://discuss.elastic.co/u/deeps)
#### Post date: [September 19, 2019, 6:19pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/5 "2019-09-19T18:19:53Z")
@Badger thanks for this info!
I ended up sending all the topics data to one index per the request from Dev's for easy searching.
Notes about the multiple pipelines was also easy to understand and implement.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 17, 2019, 6:23pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878/6 "2019-10-17T18:23:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.