# Regarding Deduplication

**URL:** <https://discuss.elastic.co/t/regarding-deduplication/180121>\
**Category:** Beats\
**Created:** [May 8, 2019, 8:07am UTC](https://discuss.elastic.co/t/regarding-deduplication/180121 "2019-05-08T08:07:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [May 8, 2019, 8:07am UTC](https://discuss.elastic.co/t/regarding-deduplication/180121/1 "2019-05-08T08:07:57Z")

</div>

Hi,  
I'm looking at avoiding duplicated entries when indexing logs indexed to elasticsearch from filebeat via logstash. To do that I'll be using the logstash `fingerprint` module...

I'd like to use more than the message field to ensure uniqueness of generated 'ids' without using the @timestamp field sent by filebeat. Given that, could I use `offset` alongside `message` to ensure unique id generation?

Regards,  
D

---

<div class="post-metadata">

**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\
**Post date:** [May 8, 2019, 9:12am UTC](https://discuss.elastic.co/t/regarding-deduplication/180121/2 "2019-05-08T09:12:49Z")

</div>

Hi @dawiro

offset + message + some datetime range should probably do it

without some datetime range there would be a chance of removing valid logs if you are harvesting a rotating log that will repeat the offset, and might re-send the same message at the same offset at a different time

that said, afaik you shouldn't be getting duplicates from filebeat unless there are problems ACKing from the output to filebeat, or some other cases at edgy scenarios. If you are receiving such amount of dupes that you need to filter them, probably there is an originating issue behind it we should be considering.

---

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [May 8, 2019, 9:22am UTC](https://discuss.elastic.co/t/regarding-deduplication/180121/3 "2019-05-08T09:22:38Z")

</div>

Hi @pmercado,

The problem with keying on timestamp is that I can't be sure filebeat isn't generating a timestamp for some portion of logs on transmission. So, in scenarios where filebeat resends logs that would be a problem.

Regarding duplicate transmits from filebeat we have seen retransmits where the logstash input flaps as the pipeline to elasticsearch blocks and becomes available again in rapid succession. It's an edge case I'm trying to account for.

Regards,  
D

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2019, 11:22am UTC](https://discuss.elastic.co/t/regarding-deduplication/180121/4 "2019-06-05T11:22:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.