Keep track of changes in documents after reindex

Elastic Stack Elasticsearch
1

Hi,

We are using logstash to reindex from an Elasticsearch 2.4 cluster to an Elasticseearch 5.5 one.

It takes 12h to copy all the documents using 10 pipelines on logstash. Pretty fine.

The problem is that our documents lie all in the same index and they can be updated or deleted. So, after 12h and knowing that scroll is a snapshot, how can we apply the changes in the documents that happened during this time?

2

There's nothing native to Elasticsearch that can handle this, you would need to do something like compare document versions in some external code.

3

Thanks, Mark!

I guessed it would be manageable only via external tools, but tried this channel just hoping for someone with a related problem.

Best,
Pedro.

4

We are working towards a changes API off the back of sequence numbers, but I don't know the ETA of that/

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.