# Logstash duplicating log events?

**URL:** <https://discuss.elastic.co/t/logstash-duplicating-log-events/36933>\
**Category:** Logstash\
**Created:** [December 10, 2015, 11:54pm UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933 "2015-12-10T23:54:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zombie101](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zombie101/32/6573_2.png) [@zombie101](https://discuss.elastic.co/u/zombie101)\
**Post date:** [December 10, 2015, 11:54pm UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933/1 "2015-12-10T23:54:12Z")

</div>

hi -- I have a fairly simple logstash setup:

- logstash runs on a web server, and sends events to redis
- logstash running on another server takes the events from redis and outputs them into ES.

I am occasionally, but not infrequently, seeing duplicated events showing up in ES. If I look at the log file for the web server, there is only one event but it shows up in ES twice, and each ES document has a different \_id. I know the events are duplicates because many of these events have unique IDs in a field specific to the application doing the logging -- the IDs are not sufficient for ES \_ids, so I don't use them as such.

The duplicates do not seem to be associated with restarts of logstash, redis, or ES. I am occasionally seeing problems with one server or another connecting to my redis instance, but these problems don't always coincide with the duplicates.

I am running logstash 2.0 and ES 2.0 (I plan on upgrading, but I don't think that this will help this particular problem).

Thanks,  
John Ouellette

---

<div class="post-metadata">

**Author:** ![zombie101](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zombie101/32/6573_2.png) [@zombie101](https://discuss.elastic.co/u/zombie101)\
**Post date:** [February 22, 2016, 12:38am UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933/2 "2016-02-22T00:38:51Z")

</div>

I was about to send in another message about this, having forgotten that I'd sent in this a couple months ago... A little more info:  
I am running Logstash 2.1.1 with the following config:

On the host generating the log events:

output {  
redis {  
host =\> ["host1", "host2"]  
data\_type =\> "list"  
key =\> "apache-raw"  
}  
}

On the host doing the indexing:

input {  
redis {  
host =\> "host1"  
data\_type =\> "list"  
key =\> 'apache-raw'  
}  
}  
input {  
redis {  
host =\> "host2"  
data\_type =\> "list"  
key =\> 'apache-raw'  
}  
}

In Elasticsearch (2.1.1) I am seeing duplicate events. The only thing I can think of is that the redis output is sending the events to each host listed in the host parameter. Admittedly, the docs don't say that it _won't_ do this, but I recall reading that it would send all events to only one of the hosts, failing over to the other host only when it fails to connect to the first host.

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [February 22, 2016, 2:02am UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933/3 "2016-02-22T02:02:11Z")

</div>

I'm no redis expert but if those two hosts are in a cluster then you are  
getting duplicate events because of the two inputs. I have no idea how it  
is recommended to read from the two hosts using the input. Take one input  
out and see if the duplication ceases.

---

<div class="post-metadata">

**Author:** ![zombie101](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zombie101/32/6573_2.png) [@zombie101](https://discuss.elastic.co/u/zombie101)\
**Post date:** [February 22, 2016, 8:39pm UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933/4 "2016-02-22T20:39:16Z")

</div>

Thanks, @Joe_Lawson -- they aren't in a cluster, but I will try removing an input.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:10am UTC](https://discuss.elastic.co/t/logstash-duplicating-log-events/36933/5 "2017-07-06T05:10:15Z")

</div>