Cluster upgrade?

Jerome_Gagnon · October 12, 2012, 5:25pm

Hi everyone,

I was just wondering how you guys proceed to upgrade ElasticSearch versions
? I'm thinking about having 2 identical clusters and upgrading one at a
time to avoid downtime... since the protocol is binary you can't really
just stop 1 node at a time and restart it, i'm I right ? i just wanted to
know how to do it, I saw a method somewhere involving firewall and things
like that, but I don't want to go there yet. I want to have 0 downtime as
you may have guessed

Greetings,

Jerome Gagnon

--

ppearcy · October 12, 2012, 8:25pm

This really depends on how much data you're dealing with and if it can all
fit onto a single node.

To summarize my setup:

Have ~400GB of data
3 production clusters, 4 nodes each (each nodes w/ 1.2TB of disk), each
cluster in different DC

So, all my data can fit on a single node, which helps.

There are two methods we use. The easier way is to move traffic from one of
the DCs and restart the cluster. Not everyone has this option, but it is
really the easiest way to go. You can typically get things back into a
yellow state in under a minute, so the downtime is quite minimal.

The other way allows us to keep things up... but we still move traffic from
the DC during some steps because of performance degradation when shards
move around. Anyways this method is:

Remove a node from the cluster
Install new s/w there and ensure it isn't getting any search traffic
Rebuild our data to the new single node cluster. We rebuild from our
primary datastore using some inhouse software that replicates any
updates/adds/deletes to both.
Move nodes from old to new cluster
We'll often leave one node running the old cluster for a short period to
allow us to roll back if necessary.

So, a lot of this is depends on how you're ingesting data into your cluster
and if all your data can fit on a subset of nodes.

Hope this helps,
Paul

On Friday, October 12, 2012 11:25:00 AM UTC-6, Jérôme Gagnon wrote:

Hi everyone,

I was just wondering how you guys proceed to upgrade Elasticsearch
versions ? I'm thinking about having 2 identical clusters and upgrading one
at a time to avoid downtime... since the protocol is binary you can't
really just stop 1 node at a time and restart it, i'm I right ? i just
wanted to know how to do it, I saw a method somewhere involving firewall
and things like that, but I don't want to go there yet. I want to have 0
downtime as you may have guessed

Greetings,

Jerome Gagnon

--

Tanguy1 · October 15, 2012, 8:16am

Hi Jerome,

Clinton wrote a nice article on how he proceed to upgrade an Es cluster:

https://groups.google.com/d/topic/elasticsearch/aB8GbaYIuqE/discussion

gist.github.com

https://gist.github.com/clintongormley/3888120

gistfile1.md

Yesterday I upgraded our running elasticsearch cluster on a site which serves a few million search requests a day, with zero downtime. I've been asked to describe the process, hence this blogpost.

To make it more complicated, the cluster was running elasticsearch version 0.17.8 (released 6 Oct 2011) and I upgraded it to the latest 0.19.10. There have been 21 releases between those two versions, with a lot of functional changes, so I needed to be ready to roll back if necessary.

Our setup:
----------

 * *elasticsearch* 
   
   We run elasticsearch on two biggish boxes: 16 cores plus 32GB of RAM. All indices have 1 replica, so all data is stored on both boxes (about 45GB of data). The primary data for our main indices is also stored in our database.  We have a few other indices whose data is stored only in elasticsearch, but are updated once daily only. Finally, we store our sessions in elasticsearch, but active sessions are cached in memcached.

This file has been truncated. show original

-- Tanguy
Twitter: @tlrx

Le vendredi 12 octobre 2012 19:25:00 UTC+2, Jérôme Gagnon a écrit :

Hi everyone,

I was just wondering how you guys proceed to upgrade Elasticsearch
versions ? I'm thinking about having 2 identical clusters and upgrading one
at a time to avoid downtime... since the protocol is binary you can't
really just stop 1 node at a time and restart it, i'm I right ? i just
wanted to know how to do it, I saw a method somewhere involving firewall
and things like that, but I don't want to go there yet. I want to have 0
downtime as you may have guessed

Greetings,

Jerome Gagnon

--

Jerome_Gagnon · October 15, 2012, 12:40pm

Yes, I saw that, so that was basically my point, since all of my data
cannot fit in one node, I will need 2 mirror clusters to complete the
process

On Monday, October 15, 2012 4:16:32 AM UTC-4, Tanguy wrote:

Hi Jerome,

Clinton wrote a nice article on how he proceed to upgrade an Es cluster:

https://groups.google.com/d/topic/elasticsearch/aB8GbaYIuqE/discussion
Upgrading a running elasticsearch cluster · GitHub

-- Tanguy
Twitter: @tlrx
tlrx (Tanguy Leroux) · GitHub

Le vendredi 12 octobre 2012 19:25:00 UTC+2, Jérôme Gagnon a écrit :

Hi everyone,

I was just wondering how you guys proceed to upgrade Elasticsearch
versions ? I'm thinking about having 2 identical clusters and upgrading one
at a time to avoid downtime... since the protocol is binary you can't
really just stop 1 node at a time and restart it, i'm I right ? i just
wanted to know how to do it, I saw a method somewhere involving firewall
and things like that, but I don't want to go there yet. I want to have 0
downtime as you may have guessed

Greetings,

Jerome Gagnon

--

Topic		Replies	Views
Update elasticsearch cluster hardware with no downtime Elasticsearch	3	1361	July 5, 2017
Upgrading from 1.3.1 -> latest version (2.1.0) Elasticsearch	2	463	July 5, 2017
Rolling Upgrade? Elasticsearch	3	744	July 6, 2017
Elastic search upgrade Elasticsearch	5	600	April 2, 2022
Upgrade procedure of elasticsearch from 1.x to 2.x Elasticsearch	2	938	July 5, 2017

Cluster upgrade?

Related topics