Elasticsearch: 2-node cluster with failover


(Prakhar Mishra) #1

I am using Elasticsearch 1.5.2 and trying to setup a 2-node cluster. These 2 nodes are primarily for failover strategy (if any one node goes down, the other one is still there to handle requests), I don't need to divide primary shards or something like that, (total data is no more than 500mb on hard-disk).

Everything goes well, until Split Brains thing kicks in. Now, since I don't have much data, I don't feel any requirement of 3 nodes. And I want to have failover mechanism too. Which means, discovery.zen.minimum_master_nodes cannot be more than 1.

Now, I have two questions:

  1. Is there any configuration possible, which could overcome 2 master nodes or Split Brains problem?
  2. If not, what all other options do I have to make it work? Like, keeping both in different clusters (one online, other one offline) and updating offline with online, time to time, for the time when online cluster goes down. Or, do I have to go for 3-node cluster?

I am going on production environment. Please help.

Env Info: Two machines (VMWare driven), having 4 cores, 8gb RAM and CentOS. We are firing 7-8 queries (1 _search and other _count) per search. Indexing op won't be very frequent (until we sync existing indices from DB through _bulk API), but searching is going to be quite frequent.


(David Pilato) #2

Add another node (master only : data false) with minimal HEAP and set minimum master nodes to 2
or
set minimum master nodes to 2 but in that case your cluster will stop working in case of failure of any node or network failure

--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 28 mai 2015 à 10:40, prakhar prakhar.mishra1990@gmail.com a écrit :

I am using Elasticsearch 1.5.2 and trying to setup a 2-node cluster. These
2 nodes are primarily for failover strategy (if any one node goes down, the
other one is still there to handle requests), I don't need to divide primary
shards or something like that, (total data is no more than 500mb on
hard-disk).

Everything goes well, until Split Brains thing kicks in. Now, since I
don't have much data, I don't feel any requirement of 3 nodes. And I want to
have failover mechanism too. Which means,
/discovery.zen.minimum_master_nodes/ cannot be more than /1/.

Now, I have two questions:

  1. Is there any configuration possible, which could overcome 2 master
    nodes
    or *Split Brains *problem?
  2. If not, what all other options do I have to make it work? Like, keeping
    both in different clusters (one online, other one offline) and updating
    offline with online, time to time, for the time when online cluster goes
    down. Or, do I have to go for 3-node cluster?

I am going on production environment. Please help.

Env Info: Two machines (VMWare driven), having 4 cores, 8gb RAM and CentOS.
We are firing 7-8 queries (1 /_search/ and other /_count/) per search.
Indexing op won't be very frequent (until we sync existing indices from DB
through /_bulk/ API), but searching is going to be quite frequent.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1432802400465-4074949.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/42DC41A9-8F90-4BB9-BBBC-35D57105A809%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Prakhar Mishra) #3

Thanks for your reply.

If I add one master only node on one of the two existing servers (2 nodes on the same physical server), with HEAP, lets say, 100mb; will that work? Or I have to add another machine for that?


(Mark Walkom) #4

That's not enough heap, you want 1GB minimum.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 28 May 2015 at 20:25, prakhar prakhar.mishra1990@gmail.com wrote:

Thanks for your reply.

If I add one master only node on one of the two existing servers (2 nodes
on the same physical server), with HEAP, lets say, /100mb/; will that
work? Or I have to add another machine for that?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949p4074961.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1432808731045-4074961.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZLP5A1k80t-D61CKLZNZ%3D8PdP4w8tsA7_O4eBWdYq4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Magnus Bäck) #5

On Thursday, May 28, 2015 at 12:25 CEST,
prakhar prakhar.mishra1990@gmail.com wrote:

If I add one master only node on one of the two existing servers (2
nodes on the same physical server), with HEAP, lets say, /100mb/;
will that work? Or I have to add another machine for that?

That should work, but it obviously means that if the machine with two
Elasticsearch nodes is taken out your cluster will be inoperable (but
it will survive if the one-node machine dies).

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150528110806.GA25289%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.


(Prakhar Mishra) #6

Sorry to bother you @Mark, but this term appropriate heap size always haunts me. Suppose, I have 2 virtual machines (4 core, 8 gb RAM each).
According to elasticsearch's documentation, half of RAM should be allocated to elasticsearch. So, this is what I came up with:

Node1: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server1)
Node2: Elasticsearch 1.5.2 Master-Only(no data) node with 1 gb HEAP (server1)
Node3: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server2)

Do you think this is a sane strategy? Moreover, if I make Master-Only node (Node2) invisible to load balancer i.e. I simply won't include it at all, will it still require 1 gb HEAP?
Please forgive me if I am asking something too naive.


(Mark Walkom) #7

Master only nodes don't need to leverage FS caching as they hold no data.
So you can easily increase their heap to 75% of system.
However you really don't want to have less than 1GB of heap irrespective of
the role it plays.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 28 May 2015 at 23:11, prakhar prakhar.mishra1990@gmail.com wrote:

Sorry to bother you @Mark, but this term appropriate heap size always
haunts me. Suppose, I have 2 virtual machines (4 core, 8 gb RAM each).
According to elasticsearch's documentation, half of RAM should be allocated
to elasticsearch. So, this is what I came up with:

Node1: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server1)
Node2: Elasticsearch 1.5.2 Master-Only(no data) node with 1 gb HEAP
(server1)
Node3: Elasticsearch 1.5.2 Master-Data node with 4 gb HEAP (server2)

Do you think this is a sane strategy? Moreover, if I make Master-Only
node
(Node2) invisible to load balancer i.e. I simply won't include it at all,
will it still require 1 gb HEAP?
Please forgive me if I am asking something too naive.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-2-node-cluster-with-failover-tp4074949p4074968.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1432818693390-4074968.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_Oj_d3qG%3DQoTW0A_ChjdXigoj2Rn2iOQurnGpCcU21dQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8