# How to find or query duplicate offsets?

**URL:** <https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456>\
**Category:** Kibana\
**Created:** [September 14, 2022, 7:26pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456 "2022-09-14T19:26:07Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [September 14, 2022, 7:26pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/1 "2022-09-14T19:26:07Z")

</div>

I’m having duplicate records in my indexes. How can I find list of duplicate records ?

Duplicate records have same offset, can you suggest the query to find list of offset with more than one count ?

Or any other way to find duplicate records

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 15, 2022, 5:38am UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/2 "2022-09-15T05:38:28Z")

</div>

What do you mean by offset?

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [September 16, 2022, 8:22pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/3 "2022-09-16T20:22:27Z")

</div>

log.offset field in Kibana. I see that’s few records are duplicate in Kibana with same log.offset value.

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [September 19, 2022, 6:03am UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/4 "2022-09-19T06:03:37Z")

</div>

> [@connectgeeks](#):
>
> log.offset field in Kibana. I see that’s few records are duplicate in Kibana with same log.offset value.

 ![Screenshot 2022-09-19 at 11.32.55 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/8/6897f34d31bd773a0df1622ad7caab586e82df02.png)

Setup breif: Filebeat - \> 2 Logstash -\> Elastic

```auto
#Filebeat output
output [host1:5044,host2:5044]
loadbalacer: true

```

This happens only for a few records like 100-500 in 1 Million. How can I fix the existing and avoid it?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 19, 2022, 9:40am UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/5 "2022-09-19T09:40:16Z")

</div>

You can use the fingerprint filter in Elasticsearch to create your own Elasticsearch document `_id` based on the timestamp and the offset, this means duplicates will update the original.

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [September 19, 2022, 12:38pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/7 "2022-09-19T12:38:20Z")

</div>

Why there is a duplication issue in the first place? And how about finding the existing duplicate data ? Can you suggest the query to find "log.offset" count \> 1

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 19, 2022, 1:04pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/8 "2022-09-19T13:04:33Z")

</div>

Logstash tries to delivery every document at least once, in some cases it may duplicate the data and this behavior is expected.

To avoid duplicate events in elasticsearch you need to use a custom `_id` instead of letting elasticsearch choose the value for the `_id` field.

This can be done using an id of your documents, if it exists, or create one based in one or more fields of the documents using the `fingerprint` filter.

Check [this blog post](https://www.elastic.co/blog/logstash-lessons-handling-duplicates) and [this blog post](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) for tips on how to deal with duplicates in Logstash and Elasticsearch.

To find the duplicate events you need to ruin an aggregation query.

Something like this:

```auto
GET your-index/_search
{
  "size": 0,
  "aggs": {
    "duplicates": {
       "terms": {
         "field": "log.offset",
         "min_doc_count": 2
       }
     }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2022, 1:05pm UTC](https://discuss.elastic.co/t/how-to-find-or-query-duplicate-offsets/314456/9 "2022-10-17T13:05:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.