Dec 1st, 2017: [EN][Elastic Stack] Upgrading to 6.0.0!


(Mark Walkom) #1

The Elastic Stack 6.0.0 was recently released as GA, and there are heaps of amazing changes, improvements, and new functionality ready to use! We've put a massive amount of work to make this the best possible release for our users (check the blog posts to dive into the details). So, let's run through a brief overview of how to upgrade the Elastic Stack — Elasticsearch, Logstash, Kibana and Beats — from any release in the 5.X series, so you can take advantage of the 6.0 goodness.

The Upgrade Process

Zero downtime, major version upgrades of Elasticsearch are here!

The first and best recommendation we have is to upgrade to the latest 5.6.X release, which is 5.6.4 as of writing. Once you are on 5.6.X, you can do rolling upgrades to 6.X, which is just as amazing as it sounds :smiley:

Now you are on 5.6.X, you can use the new and improved Upgrade Assistant, formerly known as the Migration Plugin, in Kibana. This is available as part of X-Pack Basic (all the way through to Platinum) and provides a number of features to help you with the upgrade process:

  • Cluster Checkup: runs a series of checks against your cluster and indices and generates a report identifying any issues that you need to resolve, such as indices that may be from older versions of Elasticsearch
  • Reindex Helper: graphically guides you through the process of reindexing anything that the Cluster Checkups may identify as needing to be upgraded
  • Deprecation Logging: helps identify any functional changes that may impact your use of Elasticsearch and let you catch things before upgrading to 6.0 (can be toggled on and off via the UI)

A Basic license for X-Pack provides ongoing features like Monitoring for your Elasticsearch, Kibana, and Logstash instances, along with the ability to export data to CSV (in 6.X) and 18 zoom levels for maps, all via Kibana!

A basic license is free and you can register for one right here.

Once you have resolved anything outstanding found by the Upgrade Assistant, and you have read the breaking changes documentation for each of the Elastic Stack products you use, you can move onto upgrading.

The next step it to always make sure you have a snapshot of your data, and to test any upgrade process in a non-production/critical environment first. You don’t want to run into problems halfway through an upgrade that has no back-out plan.

Now we can upgrade the Elastic Stack, in the following order:

  1. Elasticsearch
  2. Kibana
  3. Logstash
  4. Beats

Beats and Logstash that are running on 5.X are compatible with Elasticsearch 6.0.0, so you can do these in a second phase, which reduces the amount of change you need to do at once.

We have a great page on upgrades in the documentation that goes into more detail on all of the above, and it's definitely required reading.

Alternative Upgrade Path - Blue/Green

If you have the infrastructure to support it, a blue/green upgrade (or deployment) may be the easiest way forward. This essentially involves spinning up a new 6.0 Elasticsearch cluster and running a reindex from remote.
It sounds expensive in time, effort, and resources — but it has some strong benefits:

  • Reindexing allows you to leverage improved Lucene data structures, which are beneficial for resource optimisation.
  • It allows you to review your mappings (and templates) to update them to changes in data types (e.g., not_analyzed to keyword), which makes future upgrades smoother.
  • Provides an environment to test your applications on, as you can keep production running and test on the new cluster.
  • You end up with a near-zero downtime upgrade!
  • You have a safe back out option in case something doesn't go to plan!!

This method can be super effective, and we have awesome community members like Fred de Villami doing this for very large clusters (130TB) in relatively short timeframes (20 hours).

Final Thoughts

While these upgrade instructions should help make things as pain-free as possible, we still recommend taking best practice precautions. You should always make sure you have a snapshot of your data, and to test any upgrade process in a non-production/critical environment first.

The Elastic Stack, like most modern software, is an evolving platform, and it's important to stay up-to-date with releases. This means approaching thing with a mind towards continuous improvement — instead of doing a once-a-year massive upgrade, you can make smaller, less impactful changes more frequently.

Your users, developers, customers, boss, and (maybe most importantly) yourself will thank you for it!

NOTE

A late addition. Our awesome docs team (@debadair in particular) has built a fantastic page to help guide you to the right area of the docs based on your environment. It's like a choose-your-own-adventure for the Elastic Stack!

You can check it out here - https://www.elastic.co/products/upgrade_guide


(jean-pierre paris) #2

That's really a good start for 2017 Advent (Elastic) Calendar! Short and full of infos, hints...
Really pleased also to see reference to the community outside elastic!
Will articles be published during week-ends ?


Dec 8th, 2017: [CN][Elasticsearch] 使用 reindex API 迁移数据到 6.x 集群
(David Pilato) #3

Yes they will.


(Mark Walkom) #4

(I've added this to the original post, but also putting it in here in case anyone has subscribed to the category.)

A late addition. Our awesome docs team (@debadair in particular) has built a fantastic page to help guide you to the right area of the docs based on your environment. It's like a choose-your-own-adventure for the Elastic Stack!

You can check it out here - https://www.elastic.co/products/upgrade_guide


(Damien Alexandre) #5

Nice article!

One thing that's also very important when upgrading and not written here is plugins. If you rely on plugins, you will have to remove them and install them again for the new version.

And of course if your plugins are not officials, you have to make sure there is a 6.x version before anything else :slight_smile:


(Elvar) #6

Worth noting as well, even though you only keep x days of indices and you have always upgraded Elasticsearch so new indices are created with the new versions. Never a problem really, then you upgrade expecting everything to work.

Then you find out that your Kibana indice is from version 2.0 ....