Failed elasticsearch upgrade 1.5.2 --> 2.4.5


#1

Hi All,

I'm a relative newbie to RELK and it's been along to day to boot, so bear with me if I'm missing something obvious or if this question has been answered here already.

I'm attempting to upgrade a two-node RELK cluster from 1.x to 5.x. Started by cloning one node to a test environment, created a new cluster parallel to prod and have upgraded/rolled back a handful of times with little trouble (rolled back to snapshots of the VM).

In every test upgrade I was able to first upgrade ES to 2.5.4, start it, install the migration plugin and reindex in place to 5.x compatibility and upgrade again to 5.x.

Today in production I found that ES 2.4.5 wouldn't start, with errors to the effect the indices were too ancient. The links provided led to docs for the upgrade API, but those docs said the upgrade API was deprecated and I should just upgrade to 2.x and use the migration plugin to reindex. Gah.

What am I missing? Any help is appreciated.

Randy Rue
Seattle WA


(Mark Walkom) #2

Can you elaborate a little more on the process you used here?


#3

Hi Mark, thank you for your reply.

I cloned one node of the production cluster and modified it into a new single-node cluster with a different name, then modified my production rsyslog config to send duplicate events (specific application events) to the test cluster as well. I've since been testing my upgrade against that test node/cluster, upgrading redis, elasticsearch, logstash and kibana several times successfully and rolling back to snapshots to start again. For each of those upgrades, I upgraded ES from 1.5.2 to 2.4.5, then installed the migration plugin 2.0.4 and converted the indices in place, then upgraded to 5.4.1.

I created that clone once, however, and when I had it running standalone, then snapshotted it as my "pre upgrade" checkpoint. I'm pretty sure at that point I also started with new data, and this was my mistake. I'm new to this shop but understand we've been running ES since 0.20.2, and the live data appears to include indices that are too old for 2.5.4 to even start. I'm afraid I don't have those log lines, they were overwritten when I rolled back to the VM's pre-upgrade snapshot.

Running at 1.5.2, for a long list of indices dated 2013, the migration plugin 1.19 tells me "Ancient index segments: This index was created by an old version of Elasticsearch based on Lucene 3. It no longer contains segments from Lucene 3, but it needs to be marked as upgraded. Install Elasticsearch 1.7.x and upgrade this index with the upgrade API." I believe the log lines when 2.5.4 failed to start also mentioned 1.7 and the API.

But if I click on the "API" link in those alerts, I go to a page telling me "The _upgrade API is no longer useful and will be removed. Instead, see Reindex to upgrade." Which is where I started.

To continue with the "reindex in place" approach will I need to upgrade to 1.7 first, and blow the dust off the deprecated _upgrade API? If so, I'd be grateful for any guidance on that process.

Or should I be looking at the "reindex from remote" approach? At this point I'm pretty much back to square one anyway, and even with the "reindex in place" plan, once I had the old nodes upgraded my next steps were to add new nodes with current OS's/config management and retire the old ones.

Thanks again for your help.

Life is good!


#4

OK, after a closer look at the docs I realize that the "instead see Reindex to upgrade.." reference is for 5.x. To continue with my reindex in place approach, it appears I need to convert those old indices.

Do I need to upgrade to 1.7 or can I use the 1.5.2 upgrade API? Will that get me close enough for 2.4.5 to start and then the 5.x upgrade to succeed?

Or am I better off looking at "reindex from remote" anyway?


#5

OK, I ran the 1.5.2 upgrade against a few very old indices and while a check of their status shows they appear to have been updated to 1.5.2 acceptability, the 1.19 migration plugin still flags them and says to upgrade to 1.7...


(Mark Walkom) #6

I am not sure we catered for indices from 0.X releases in the assistant to be honest :frowning:


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.