# Logstash sends multiple copies of data

**URL:** <https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267>\
**Category:** Logstash\
**Created:** [December 11, 2018, 12:46am UTC](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267 "2018-12-11T00:46:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nazgul](https://avatars.discourse-cdn.com/v4/letter/n/6de8d8/32.png) [@nazgul](https://discuss.elastic.co/u/nazgul)\
**Post date:** [December 11, 2018, 12:46am UTC](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267/1 "2018-12-11T00:46:14Z")

</div>

Hello everyone,  
need you help on next question:

I have 5 logstash.conf file to run.  
They are equal and only "Input path/.../....log" is different.

[sudo] password for maxim:

input {  
file {  
path =\> "/var/log/test.log"  
start\_position =\> "beginning"  
}  
}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "logstash3-%{+YYYY.MM.dd}"  
document\_type =\> "my\_test3\_doc"

}  
stdout {}  
}

In fact,  
logstash sends me 5 messages per input using the same data.  
The only one field is different is "\_id".

For example,  
I have 5 conf files then I receive 5 times messages for each on of them... (5x5 = 25 in total).  
If I have 20 conf files simultaneously then I will receive 20 duplicated messages with different \_id. 20 x 20 = 400 Messages in total).  
How to stop sending copies?  
Do I need to change Document\_id? I tried and not sure that correctly.  
Please advise.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 11, 2018, 3:35am UTC](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267/2 "2018-12-11T03:35:31Z")

</div>

All config files in the directory are concatenated into a single pipeline. This means that each event generated by an input plugin will go through all filters and be sent to all outputs (all 5 of them). You can get around this by creating a single config file with multiple inputs, use conditionals or use the relatively new multiple pipeline feature. This is a common misunderstanding so you should easily be able to find examples.

---

<div class="post-metadata">

**Author:** ![nazgul](https://avatars.discourse-cdn.com/v4/letter/n/6de8d8/32.png) [@nazgul](https://discuss.elastic.co/u/nazgul)\
**Post date:** [December 11, 2018, 7:07pm UTC](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267/3 "2018-12-11T19:07:00Z")

</div>

Thanks. Now i know where to dig.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2019, 7:07pm UTC](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267/4 "2019-01-08T19:07:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.