# Duplicate records when scaling logstash **URL:** \ **Category:** Logstash\ **Created:** [March 2, 2016, 1:23pm UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252 "2016-03-02T13:23:59Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Post date:** [March 2, 2016, 1:23pm UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/1 "2016-03-02T13:23:59Z")
Hi, I am using ELK stack like below LSF==\> logstash-Shipper--\>Redis--\>Indexer--\>nGinx--\>ES I have scaled shipper and indexers as below 1. CASE1 for all logs shipper.conf Indexer.conf ``` input { redis { host => "localhost" data_type => "list" key => "logstash" } } ``` 1. CASE2 for tomee access-logs Shipper-url.conf ``` output { redis { host => "localhost" data_type => "list" key => "access-logs" congestion_interval => 1 } } ``` indexer-url.conf ``` input { redis { host => "localhost" data_type => "list" key => "access-logs" } } ``` I am expecting that indexers should pick up events with specific keys mentioned in input configuration. However, its not exactly the result. There are duplicate logs I can see in Kibana. Please tell me what is the wrong I am doing? br, Sunil Chaudhari ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2016, 7:33am UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/2 "2016-03-03T07:33:33Z")
Are the shipper and indexer different instances? ---
**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Post date:** [December 27, 2016, 3:00pm UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/3 "2016-12-27T15:00:01Z")
Sorry for replying to an old post, but I would like to know how it finally worked for you. ---
**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Post date:** [December 28, 2016, 6:55am UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/4 "2016-12-28T06:55:38Z")
Hi, I reverted the configuration of 2 indexer instances, because I didn't find the root cause. However, Later I found that there were multiple processes running behind. That might be the root cause. But I didn't tried it again. br, Sunil. ---
**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Post date:** [December 28, 2016, 3:54pm UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/5 "2016-12-28T15:54:47Z")
Thanks Sunil for the response. I have shipper and indexer in one instance as @warkolm mentioned. I was testing out logstash with Redis and wrote 2 configuration files. `conf1 --> file input and Redis output, conf2 --> Redis input and Elasticsearch output.` I want to know why keeping multiple configuration files in `/etc/logstash/conf` directory duplicates events. Say, If I use different instances, one for indexer and another for shipper, this duplication doesn't happen. Also, if I ran logstash with just `conf1` file and then ran logstash again with `conf2` file, there was no duplication. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 28, 2016, 5:03pm UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/6 "2016-12-28T17:03:26Z")
Have a look at the following threads: > [@Logstash not creating right number of documents when passing folder as an argument](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720): > > Hi, What I'm doing: Created 4 conf files under a root directory called XYZ. Each of these conf files will import 1000 rows from SQL and tables being imported are unique in all 4 conf files. when ran separately the number of documents created are 1000 in each index but when running conf file with root folder as argument the count is not 1000 in indexes. I've also observed each index is picking different document. I'm also using templates for each of the index and template name is different all … > [@Logstash and multiple pipelines. Why are my pipelines merging?](https://discuss.elastic.co/t/logstash-and-multiple-pipelines-why-are-my-pipelines-merging/68920): > > Hi stashers, I have two pipelines, A & B, both configured to process separate CSV files of different formats into separate indexes on the same Elastic Cloud cluster. Something very peculiar (to me...) is happening when they are both running in the same logstash 5.0.0 (can I use higher level versions of LS with ES 5.0.0?). A input: /path/to/As/\*.csv filter: lots of csv columns, make some fields lowercase, convert fields to types, prepend "A-" to a field to use for the index name, match date o… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:29am UTC](https://discuss.elastic.co/t/duplicate-records-when-scaling-logstash/43252/7 "2017-07-06T04:29:39Z")