I am upgrading a cluster from 1.5 to 2.x. When I asked at Elasticon, I was told that we should stand up a new cluster and then import the data from the 1.5 cluster to 2.x. I have created a new ES2 cluster with a new Logstash 2.x server as well. The goal is to import all data prior to 3/1/2016 into the new cluster. Upgrade the existing 1.5 Logstash servers to 2.x. Then import the delta of the data from the old cluster.
I was told to use this blog as a reference point for a logstash.conf file, but the syntax in this sample is not recognized by Logstash 2.x as valid (http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/), so I made some adjustments:
input {
elasticsearch {
hosts => "oldclusternode1:9200"
index => "logstash-2015.10.14" # just testing a single index first to see if it works
query => "*"
size => 100
scroll => "5m"
docinfo => true
}
}
output {
elasticsearch {
hosts => "newclusternode1:9200"
index => "logstash-2015.10.14"
}
stdout {
codec => "dots"
}
}
The above passes the --configtest. But when I run it, I get Warnings in the old clusters.log file saying:
[logstash-2015.10.14][0], node[ew0rLPgVQ5yjophkhSBOOw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@55f81314] lastShard [true]
org.elasticsearch.search.SearchParseException: [logstash-2015.10.14][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [na]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:721)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:557)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:529)
at org.elasticsearch.search.SearchService.executeScan(SearchService.java:224)
at org.elasticsearch.search.action.SearchServiceTransportAction$19.call(SearchServiceTransportAction.java:486)
at org.elasticsearch.search.action.SearchServiceTransportAction$19.call(SearchServiceTransportAction.java:483)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchParseException: Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@49
at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:259)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:691)
And I see nothing in the newcluster.log file.
Can someone please tell me what I am doing wrong? I have run the migration script and it came back all green. I really need to get this single index transferred and then start doing bulk data sends to test the load on the old cluster. While I am doing this bulk migration, I am watching to ensure that I don't overload the production cluster while migrating data to the new cluster.