Updating documents with deterministic ID in older index after rollover

[Tomasz_Szymanski (Tomasz Szymański)]

Hi Guys,

I have system where I retrieve data from different sources from .NET project, put it on a rabbitmq queue, grab them from logstash, and logstash indexes these data in my elasticsearch.

Recently I introduced index-lifecycle-managment to rollover the indexes when they grow too big. Which worked fine until l encountered following issue.

My data have deterministic GUID IDs, when my index rollover there is a probability that the data that are supposed to be indexed are already in ROLLED index. This is a must, because I am updating some of these data, and their GUID IDs are generated deterministically.

So basically, I do not want data that IDs are already in old indexes to be indexed in new index, I want them updated in old appropriate index.

Do you have any suggestions on how to overcome this issue?
I can provide logstash and elasticsearch ILM policy/template configs if needed.

[DaroH]

This is an issue we encountered recently as well, but havent found satisfying solution yet. Anyone have any input on this?

[Christian_Dahlqvist]

Assuming you have a time stamp associated with each document which you can use to determine index, I would recommend using ILM without rollover.