I've been flailing for some time on upgrading a two-node ES 1.x cluster to 5.x, finally hitting a wall after upgrading the indices in place from 0.20 (some of them, anyway) to 1.7 and then upgrading as far as 2.4.5 successfully. Then I've been unable to use the migration plugin to convert the indices any further for an upgrade to 5.x.
Finally scrapped that plan and spun up a new 5.x cluster which is working beautifully, and am now flailing on importing the old indices to the new cluster. I've cloned one node of the existing old cluster (still in prod), deleted all indices and restored from a snapshot of the prod cluster, and then upgraded to 2.45. Set all indices to a single copy (no replicas). The short version is I now have a single node running a "holding" copy of the previous indices, status is green and all is well.
Note that this ELK cluster is primarily holding logs and not heavily queried so we're now running 1 shard, but the older indices have differing shard counts from 5 down to one. But I seem to be able to pull indices with any number of shards set, as long as the source is working at all (see below).
Now I'm trying to run the reindex API to pull from the holding node to the new cluster, and while I can pull a few successfully, after a few indices the source system bogs down and eventually ES stops responding. In fact, it stops responding to any calls including _cat/indices, _cluster/health, _tasks, anything. And if I run those too often, they'll crash it too. Eventually I have to restart ES, then wait for all unassigned shards to be assigned, about 3200 shards and 45 minutes.
This is running on a 4 core VM with 8GB RAM, BTW.
I admit I'm pretty new to ELK in general. Am I missing something obvious?
Hope to hear from you,
Randy in Seattle