# LS posts same document to ES multiple times
**URL:** https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197
**Category:** Logstash
**Created:** [January 30, 2017, 11:37am UTC](https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197 "2017-01-30T11:37:15Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@dtrizna](https://discuss.elastic.co/u/dtrizna)
#### Post date: [January 30, 2017, 11:37am UTC](https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197/1 "2017-01-30T11:37:15Z")
Hi, community!
My ELK stack has some strange behavior.
Recently I found, that every document in ES is indexed multiple times.
Documents are Netflow flows.
They have different \_id values, although it is the same document (describing same connection).
I've proved that data is received correct (on udp socket), but bulk requests to ES already sent this document multiple times (in different requests). So the problem is within LS.
After some data fetching and configuration tuning I observed, that document is multiplied 5 times. The data boost appeared after I added additional config files under /etc/logstash/conf.d/, and exactly the same number of configuration files I have there. Just for test pusposes I removed 2 of them, and yes - number of document copies appeared to be 3.
How it may be? Have anyone faced same symptoms?
I can provide any necessary info from Logstash socket, please ask.
OS: Centos 7; Linux 3.10.0-514.2.2.el7.x86\_64
LS: "version" : "5.1.2"
One of configuration files:
input {
udp {
port =\> 2056
codec =\> netflow { }
}
}
output {
elasticsearch {
pool\_max =\> 1000
pool\_max\_per\_route =\> 400
manage\_template =\> false
flush\_size =\> 10000
hosts =\> localhost
index =\> "netflow-%{+YYYY.MM.dd.HH}"
}
}
Within logstash.yml I changed only: pipeline.batch.size: 20000
Nr. of workers is equal to CPU - 24, output workers = 1
Any ideas?
Thanks,
Dmitry
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 30, 2017, 11:41am UTC](https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197/2 "2017-01-30T11:41:49Z")
Configuration files in a directory are concatenated, not run separate in isolation within a Logstash instance, so if you have multiple outputs configured, all events will be sent to all outputs unless you are using conditionals to control this.
---
### Author:  [@dtrizna](https://discuss.elastic.co/u/dtrizna)
#### Post date: [January 30, 2017, 11:52am UTC](https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197/3 "2017-01-30T11:52:28Z")
Yeah, now I see, how obvious is this!
I've seen constructions like this:
if ( [host] = "10.1.1.1" ) {
elasticsearch {
index =\> "netflow-%{+YYYY.MM.dd}" } }
This is what you talked about?
Doesn't such logical operations slow the forwarding process?
Unfortunately I've didn't seen info about file concatenation in Oficial documentation (probably, simply doesn't found it).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 27, 2017, 11:52am UTC](https://discuss.elastic.co/t/ls-posts-same-document-to-es-multiple-times/73197/4 "2017-02-27T11:52:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.