Re-Indexing Via ES Document Numbers

Elastic Stack Logstash
1

Hi,

I am trying to re-index some data by reading the information out of ES using an input of:-

input {
	elasticsearch {
    	    hosts => "<our_host_name>"
    	    index => "<target_index_name>"
     	}
}

This appears to be pulling down all the documents in the index.

I am then executing some very simple transformations on some of the fields. Nothing major and no errors occurring there.

I then wish to send the data back to ES into a new index. Using the same @timestamp values as the original events the only difference being the slightly modified fields and values. This seems to work as the new index is created with all the events... but....

The original index contains around 5 million events. The new index just keeps on growing! Over 17 million events were pushed into the new index before I stopped it.

Am I missing something in the re-index process? Apologies if this has been asked before.