Updating documents with deterministic ID in older index after rollover

Elastic Stack Elasticsearch
1

Hi Guys,

I have system where I retrieve data from different sources from .NET project, put it on a rabbitmq queue, grab them from logstash, and logstash indexes these data in my elasticsearch.

Recently I introduced index-lifecycle-managment to rollover the indexes when they grow too big. Which worked fine until l encountered following issue.

My data have deterministic GUID IDs, when my index rollover there is a probability that the data that are supposed to be indexed are already in ROLLED index. This is a must, because I am updating some of these data, and their GUID IDs are generated deterministically.

So basically, I do not want data that IDs are already in old indexes to be indexed in new index, I want them updated in old appropriate index.

Do you have any suggestions on how to overcome this issue?
I can provide logstash and elasticsearch ILM policy/template configs if needed.

2 Likes
2

This is an issue we encountered recently as well, but havent found satisfying solution yet. Anyone have any input on this?

3

Assuming you have a time stamp associated with each document which you can use to determine index, I would recommend using ILM without rollover.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.