# Duplicate entries in elastic for the same message

**URL:** <https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732>\
**Category:** Logstash\
**Created:** [March 15, 2023, 7:51am UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732 "2023-03-15T07:51:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![aks03](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@aks03](https://discuss.elastic.co/u/aks03)\
**Post date:** [March 15, 2023, 7:51am UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732/1 "2023-03-15T07:51:54Z")

</div>

Hey everyone, I have been trying to fix this error of duplicate entries into Elasticsearch through logstash  
Below is the example of the two entries  
Entry 1:  
{  
"\_index": "test-index",  
"\_type": "doc",  
"\_id": "3044350386",  
"\_version": 2,  
"\_score": 1,  
"\_source": {  
"message": "same message",  
"log\_type": "registration log",  
"@version": "1",  
"path": "eg.log",  
"@timestamp": "2023-03-15T07:43:17.019Z",  
"host": "same host"  
}  
}  
Entry 2:  
{  
"\_index": "test-index",  
"\_type": "doc",  
"\_id": "nEg55IYBsSh0OlkJJ8nh",  
"\_version": 1,  
"\_score": 1,  
"\_source": {  
"@version": "1",  
"@timestamp": "2023-03-15T07:43:16.335Z",  
"path": "eg.log",  
"host": "same host",  
"message": "same message"  
}  
}  
I have tried using fingerprint also but still there are two entries one with the id from fingerprint and other is the autogenerated id  
Any help would be appreciated

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 15, 2023, 1:20pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732/2 "2023-03-15T13:20:00Z")

</div>

> [@aks03](#):
>
> I have been trying to fix this error of duplicate entries into Elasticsearch through logstash

Please share your Logstash configuration.

---

<div class="post-metadata">

**Author:** ![aks03](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@aks03](https://discuss.elastic.co/u/aks03)\
**Post date:** [March 15, 2023, 4:57pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732/3 "2023-03-15T16:57:13Z")

</div>

Thanks for the reply. The problem was related to multiple logstash processes, after I killed the processes the duplication was stopped

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [March 16, 2023, 8:58am UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732/4 "2023-03-16T08:58:46Z")

</div>

Duplicates data can be handle by finderprint. Check [the blog](https://www.elastic.co/blog/logstash-lessons-handling-duplicates).  
However, as Leandro said, share .conf and data sample in case you still have problems.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2023, 8:58am UTC](https://discuss.elastic.co/t/duplicate-entries-in-elastic-for-the-same-message/327732/5 "2023-04-13T08:58:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.