Using fingerprint and document_id for at-least-once delivery and dedupe with ILM

Elastic Stack Logstash
1

TL;DR

Is ILM compatible with at-least-once-delivery / deduplication / idempotence?

Longer version..

I routinely use a fingerprint filter combined with the elasticsearch output's document_id setting for at-least-once delivery.

Using named indices (with date math), this had some nice properties, especially around the ability to replay logs in the event of partial ingest or changed processing logic, without worrying about duplicates ending up in my indices. There was always a maximum of one document for any given _id, and the same source data always ended up in the same ES index.

But now that I'm moving to ILM, this no longer works, because I can't rely on the document going into the same index.

Is this just a known limitation that I have to accept and deal with? Or is there a way to make the elasticsearch output quasi-idempotent even with ILM enabled?

2

ILM is built on the idea that the indices are more-or-less read only once they have been rolled.

If you want to update the old data, you need to talk to the index it lives in rather than the ILM alias, which is outside the scope of what ILM is designed to do.

3

Thanks Mark.

So I'm 100% clear - there is no way to get the benefits of ILM and have at-least-once pipelines as discussed above - correct?

4

I see this has been discussed at some length on github.

It's clear that, at this time, I can't have my cake and eat it too.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.