# Logstash handling duplicates using UUID

**URL:** <https://discuss.elastic.co/t/logstash-handling-duplicates-using-uuid/147215>\
**Category:** Logstash\
**Created:** [September 4, 2018, 1:30pm UTC](https://discuss.elastic.co/t/logstash-handling-duplicates-using-uuid/147215 "2018-09-04T13:30:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\
**Post date:** [September 4, 2018, 1:30pm UTC](https://discuss.elastic.co/t/logstash-handling-duplicates-using-uuid/147215/1 "2018-09-04T13:30:54Z")

</div>

I have requirement to use a mechanism to avoid duplicates on events received from log data. These duplicates are being caused by ES or Logstash indexer crashes.

I was going through this link:

[https://www.elastic.co/blog/logstash-lessons-handling-duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)

Looks like UUID option works well for us as we use a queue in between logstash shippers and indexers.  
But, since we have multiple logstash shippers (producers) and each generates this UUID, is it possible that we run into issues where same UUID is generated by 2 or more shippers (producers) ? If this happens, we might be updating some existing events, even though unintended.

Please advise. Thanks !!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2018, 1:31pm UTC](https://discuss.elastic.co/t/logstash-handling-duplicates-using-uuid/147215/2 "2018-10-02T13:31:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.