Keep track of changes in documents after reindex


(Pedro Brentan) #1

Hi,

We are using logstash to reindex from an Elasticsearch 2.4 cluster to an Elasticseearch 5.5 one.

It takes 12h to copy all the documents using 10 pipelines on logstash. Pretty fine.

The problem is that our documents lie all in the same index and they can be updated or deleted. So, after 12h and knowing that scroll is a snapshot, how can we apply the changes in the documents that happened during this time?


(Mark Walkom) #2

There's nothing native to Elasticsearch that can handle this, you would need to do something like compare document versions in some external code.


(Pedro Brentan) #3

Thanks, Mark!

I guessed it would be manageable only via external tools, but tried this channel just hoping for someone with a related problem.

Best,
Pedro.


(Mark Walkom) #4

We are working towards a changes API off the back of sequence numbers, but I don't know the ETA of that/


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.