I am upgrading a cluster from 1.5 to 2.x. When I asked at Elasticon, I was told that we should stand up a new cluster and then import the data from the 1.5 cluster to 2.x. I have created a new ES2 cluster with a new Logstash 2.x server as well. The goal is to import all data prior to 3/1/2016 into the new cluster. Upgrade the existing 1.5 Logstash servers to 2.x. Then import the delta of the data from the old cluster.
I was told to use this blog as a reference point for a logstash.conf file, but the syntax in this sample is not recognized by Logstash 2.x as valid (http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/), so I made some adjustments:
hosts => "oldclusternode1:9200"
index => "logstash-2015.10.14" # just testing a single index first to see if it works
query => "*"
size => 100
scroll => "5m"
docinfo => true
hosts => "newclusternode1:9200"
index => "logstash-2015.10.14"
codec => "dots"
The above passes the --configtest. But when I run it, I get Warnings in the old clusters.log file saying:
[logstash-2015.10.14], node[ew0rLPgVQ5yjophkhSBOOw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@55f81314] lastShard [true]
org.elasticsearch.search.SearchParseException: [logstash-2015.10.14]: from[-1],size[-1]: Parse Failure [Failed to parse source [na]]
Caused by: org.elasticsearch.ElasticsearchParseException: Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@49
And I see nothing in the newcluster.log file.
Can someone please tell me what I am doing wrong? I have run the migration script and it came back all green. I really need to get this single index transferred and then start doing bulk data sends to test the load on the old cluster. While I am doing this bulk migration, I am watching to ensure that I don't overload the production cluster while migrating data to the new cluster.