Reindexing with logstash

Elastic Stack Logstash
1

I have a an old elastic cluster running 1.7.5 that is underperforming.
I've setup a new cluster on 2.4.6 and want to use logstash to reindex data from old to new.
In the process I am consolidating daily indexes to monthlies.

What is the behavior around the logstash elastic input? If i give it an index to feed data to the new elasticsearch does it keep looping over the index and keep feeding duplicate docs over and over again? Does it know once the index specified in the logstash input has completed?
If not what is the best way to track this?
Or does adding document_id => "%{[@metadata][_id]}" prevent this if it's the case?

Thanks.

2

Does it know once the index specified in the logstash input has completed?

No, it doesn't track this between Logstash restarts.

Or does adding document_id => "%{[@metadata][_id]}" prevent this if it's the case?

I'm not sure it's sufficient, but it should definitely help. I don't recall if the elasticsearch output requires additional options to fully support upsert operations.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.