Hot-Warm-Cold - avoid duplicates


I'm moving to a new ES cluster which also uses the Hot-Warm-Cold architecture. Now, when we imported our old data (600 million documents) it took quite a long time and after a while we noticed that we have duplicate documents, with the same _id field.

It seems like a few documents are present in both the "Hot" index as well as the "Warm" index. Previously this wasn't possible since we only had one index, but now there are separate indexes for Hot and Warm so duplicates are possible.

Are there any ways of avoiding these things happening?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.