# Using fingerprint and document\_id for at-least-once delivery and dedupe with ILM

**URL:** <https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522>\
**Category:** Logstash\
**Tags:** ilm-index-lifecycle-management\
**Created:** [June 10, 2021, 6:55am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522 "2021-06-10T06:55:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 10, 2021, 6:55am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522/1 "2021-06-10T06:55:27Z")

</div>

## TL;DR

Is ILM compatible with at-least-once-delivery / deduplication / idempotence?

### Longer version..

I routinely use a `fingerprint` filter combined with the elasticsearch output's `document_id` setting for at-least-once delivery.

Using named indices (with date math), this had some nice properties, especially around the ability to replay logs in the event of partial ingest or changed processing logic, without worrying about duplicates ending up in my indices. There was always a maximum of one document for any given `_id`, and the same source data always ended up in the same ES index.

But now that I'm moving to ILM, this no longer works, because I can't rely on the document going into the same index.

Is this just a known limitation that I have to accept and deal with? Or is there a way to make the elasticsearch output quasi-idempotent _even with ILM enabled_?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 10, 2021, 7:05am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522/2 "2021-06-10T07:05:50Z")

</div>

ILM is built on the idea that the indices are more-or-less read only once they have been rolled.

If you want to update the old data, you need to talk to the index it lives in rather than the ILM alias, which is outside the scope of what ILM is designed to do.

---

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 10, 2021, 7:10am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522/3 "2021-06-10T07:10:02Z")

</div>

Thanks Mark.

So I'm 100% clear - there is no way to get the benefits of ILM **and** have at-least-once pipelines as discussed above - correct?

---

<div class="post-metadata">

**Author:** ![tomr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomr/32/48260_2.png) [@tomr](https://discuss.elastic.co/u/tomr)\
**Post date:** [June 10, 2021, 7:21am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522/4 "2021-06-10T07:21:37Z")

</div>

I see this has been discussed at some length [on github](https://github.com/elastic/elasticsearch/issues/44794).

It's clear that, at this time, I can't have my cake and eat it too.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2021, 7:21am UTC](https://discuss.elastic.co/t/using-fingerprint-and-document-id-for-at-least-once-delivery-and-dedupe-with-ilm/275522/5 "2021-07-08T07:21:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.