I have system where I retrieve data from different sources from .NET project, put it on a rabbitmq queue, grab them from logstash, and logstash indexes these data in my elasticsearch.
Recently I introduced index-lifecycle-managment to rollover the indexes when they grow too big. Which worked fine until l encountered following issue.
My data have deterministic GUID IDs, when my index rollover there is a probability that the data that are supposed to be indexed are already in ROLLED index. This is a must, because I am updating some of these data, and their GUID IDs are generated deterministically.
So basically, I do not want data that IDs are already in old indexes to be indexed in new index, I want them updated in old appropriate index.
Do you have any suggestions on how to overcome this issue?
I can provide logstash and elasticsearch ILM policy/template configs if needed.