I am using the following pipeline to copy data from index to another.
input {
elasticsearch {
hosts => ["sourcehost:9200"]
index => "source-index"
query => '{"query":{"match_all":{}}}'
size => 500
scroll => "5m"
docinfo => true
}
}
output {
elasticsearch {
hosts => ["targethost:9200"]
index => "target-index"
manage_template => false
workers => 2
}
}
The issue with logstash is its not preserving the mapping of the source index.
For ex : an field is having following mappings in source index
"INSERT_DATE": {
"type": "date",
"store": true,
"format": "yyyy-MM-dd HH:mm:ss"
}
But in target index the mapping is
"INSERT_DATE": {
"type": "string"
}
LS has no concept of copy the mappings over.
You need to either pull the mapping down and define it in the output, or copy it over before (or use templates).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.