Hi Mark, thank you for your reply.
I cloned one node of the production cluster and modified it into a new single-node cluster with a different name, then modified my production rsyslog config to send duplicate events (specific application events) to the test cluster as well. I've since been testing my upgrade against that test node/cluster, upgrading redis, elasticsearch, logstash and kibana several times successfully and rolling back to snapshots to start again. For each of those upgrades, I upgraded ES from 1.5.2 to 2.4.5, then installed the migration plugin 2.0.4 and converted the indices in place, then upgraded to 5.4.1.
I created that clone once, however, and when I had it running standalone, then snapshotted it as my "pre upgrade" checkpoint. I'm pretty sure at that point I also started with new data, and this was my mistake. I'm new to this shop but understand we've been running ES since 0.20.2, and the live data appears to include indices that are too old for 2.5.4 to even start. I'm afraid I don't have those log lines, they were overwritten when I rolled back to the VM's pre-upgrade snapshot.
Running at 1.5.2, for a long list of indices dated 2013, the migration plugin 1.19 tells me "Ancient index segments: This index was created by an old version of Elasticsearch based on Lucene 3. It no longer contains segments from Lucene 3, but it needs to be marked as upgraded. Install Elasticsearch 1.7.x and upgrade this index with the upgrade API." I believe the log lines when 2.5.4 failed to start also mentioned 1.7 and the API.
But if I click on the "API" link in those alerts, I go to a page telling me "The _upgrade API is no longer useful and will be removed. Instead, see Reindex to upgrade." Which is where I started.
To continue with the "reindex in place" approach will I need to upgrade to 1.7 first, and blow the dust off the deprecated _upgrade API? If so, I'd be grateful for any guidance on that process.
Or should I be looking at the "reindex from remote" approach? At this point I'm pretty much back to square one anyway, and even with the "reindex in place" plan, once I had the old nodes upgraded my next steps were to add new nodes with current OS's/config management and retire the old ones.
Thanks again for your help.
Life is good!
