# Duplicate records in Kibana

**URL:** <https://discuss.elastic.co/t/duplicate-records-in-kibana/245896>\
**Category:** Logstash\
**Created:** [August 21, 2020, 9:48am UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896 "2020-08-21T09:48:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![apurvakhatri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apurvakhatri/32/69819_2.png) [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\
**Post date:** [August 21, 2020, 9:48am UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896/1 "2020-08-21T09:48:02Z")

</div>

![Kibana duplicate](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a0df4e453168562cb6ee156340bc18932480f9eb.jpeg)

I noticed that there are multiple records for the same log. They have the same clientip, time & host.

Issue: Duplicate records. (There are multiple copies of same record)

Requirement: There should only be one record of any particular log injected into elasticsearch.

Any Solutions would be of great help!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 21, 2020, 3:34pm UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896/2 "2020-08-21T15:34:18Z")

</div>

Build a [fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) based on the fields that make the record unique and use it to set the document\_id on the elasticisearch output.

---

<div class="post-metadata">

**Author:** ![apurvakhatri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apurvakhatri/32/69819_2.png) [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\
**Post date:** [August 21, 2020, 5:18pm UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896/3 "2020-08-21T17:18:23Z")

</div>

Thanks! I'm using it. It still happens but for very few number of records.  
I want to know but why?  
Logstash should send only one record, right? Filebeat also maintains state of files by keeping a record of offset in the registry.

---

<div class="post-metadata">

**Author:** ![apurvakhatri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apurvakhatri/32/69819_2.png) [@apurvakhatri](https://discuss.elastic.co/u/apurvakhatri)\
**Post date:** [September 4, 2020, 8:08am UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896/4 "2020-09-04T08:08:26Z")

</div>

Thank you!  
Can you tell me why it happens?  
Logstash should send only one record, right? Filebeat also maintains state of files by keeping a record of offset in the registry.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2020, 8:08am UTC](https://discuss.elastic.co/t/duplicate-records-in-kibana/245896/5 "2020-10-02T08:08:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.