# Duplicate Docs Getting Created

**URL:** <https://discuss.elastic.co/t/duplicate-docs-getting-created/355006>\
**Category:** Elasticsearch\
**Created:** [March 8, 2024, 6:55am UTC](https://discuss.elastic.co/t/duplicate-docs-getting-created/355006 "2024-03-08T06:55:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sajid](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sajid](https://discuss.elastic.co/u/sajid)\
**Post date:** [March 8, 2024, 6:55am UTC](https://discuss.elastic.co/t/duplicate-docs-getting-created/355006/1 "2024-03-08T06:55:10Z")

</div>

We are running fluentd in our Kubernetes cluster as DaemonSet and logs are getting shipped to Elasticsearch directly. We are observing duplicate logs are being stored in ES i.e. for a single line of log, we are seeing multiple docs. What could be the reason of the duplication?

```auto
Mar 7, 2024 @ 13:59:53.110	Received request with SearchGuid - ea1c7da0-ebf1-43e6-acf5-15d6b5ed2008, ContractId - 6, TransactionId - None at 2024-03-07 13:59:53.110201-05:00	Mar 7, 2024 @ 08:59:53.110

Mar 7, 2024 @ 13:59:53.110	Received request with SearchGuid - ea1c7da0-ebf1-43e6-acf5-15d6b5ed2008, ContractId - 6, TransactionId - None at 2024-03-07 13:59:53.110201-05:00	Mar 7, 2024 @ 08:59:53.110

Mar 7, 2024 @ 13:59:53.110	Received request with SearchGuid - ea1c7da0-ebf1-43e6-acf5-15d6b5ed2008, ContractId - 6, TransactionId - None at 2024-03-07 13:59:53.110201-05:00	Mar 7, 2024 @ 08:59:53.110

Mar 7, 2024 @ 13:59:53.110	Received request with SearchGuid - ea1c7da0-ebf1-43e6-acf5-15d6b5ed2008, ContractId - 6, TransactionId - None at 2024-03-07 13:59:53.110201-05:00	Mar 7, 2024 @ 08:59:53.110

Mar 7, 2024 @ 13:59:53.110	Received request with SearchGuid - ea1c7da0-ebf1-43e6-acf5-15d6b5ed2008, ContractId - 6, TransactionId - None at 2024-03-07 13:59:53.110201-05:00	Mar 7, 2024 @ 08:59:53.110

```

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 8, 2024, 7:34am UTC](https://discuss.elastic.co/t/duplicate-docs-getting-created/355006/2 "2024-03-08T07:34:11Z")

</div>

You'll probably need to debug within Fluentd.

Elasticsearch can duplicate docs if they're received multiple times (exactly once delivery isn't a thing), and we rely on the client to generate unique ids if duplication is a risk/problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 5, 2024, 7:34am UTC](https://discuss.elastic.co/t/duplicate-docs-getting-created/355006/3 "2024-04-05T07:34:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.