working on a project where we query es periodically and with range filters, in order to detect trends (like spikes) of certain conditions.
To make sure all indexed event settle down and we don't miss any of them, we used a 10m query delay (we query events that were indexed at least 10 minutes before they got queried)
Now we found lots of false positives because the query would miss some events during detection time, but later on appears in the result (after say 6 hours).
I thought they might got delayed in the indexing process for whatever reasons, but examining the indexing time (we enabled _timestamp field) tells that was not the case.
Any hints on how this could happen?
Thanks,
Fang