Problem of sync elasticsearch data after failover

Elastic Stack Elasticsearch
1

Hi everyone, I am new to elasticsearch. I'v setted up a small cluster (with
version 0.90.1).
Within the cluster I have 4 nodes total:

loadblancer node 1: on app1 server
loadblancer node 2: on app2 server
master node 1: stand alone server 1
master node 2: stand alone server 2

This architecture is come from the idea in this discussion:


See Duc.Duong's reply

each master node have config as:
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:
[":9300",":9300"]

I can get cluster have green status now. For me, the most important thing
is failover, and now this is basicly avaliable.
That is to say, I can shut down one master (using command "service
elasticsearch stop") and ES still work correctly, and when I start that
master again,
it joins the cluster as a new candidate master, the status goes green, and
data gets synced.

But,in extreme situation, if I did not start that master again, instead:
I shutdown the remaining master after inserting some documents, then,
I start the 2 masters, status goes green again, but the recently inserted
documents won't be synced.

How can I get these documents synced?
I'll thanks for any ideas.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I think you restarted master1 first (the one who have less documents) and then master2.
Try to do the opposite and wait for master2 to be up and running before starting master1.

Also, note that in production, you should consider having 3 master nodes.

My 2 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 mai 2014 à 03:50, cl f fclddcn@gmail.com a écrit :

Hi everyone, I am new to elasticsearch. I'v setted up a small cluster (with version 0.90.1).
Within the cluster I have 4 nodes total:

loadblancer node 1: on app1 server
loadblancer node 2: on app2 server
master node 1: stand alone server 1
master node 2: stand alone server 2

This architecture is come from the idea in this discussion:


See Duc.Duong's reply

each master node have config as:
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [":9300",":9300"]

I can get cluster have green status now. For me, the most important thing is failover, and now this is basicly avaliable.
That is to say, I can shut down one master (using command "service elasticsearch stop") and ES still work correctly, and when I start that master again,
it joins the cluster as a new candidate master, the status goes green, and data gets synced.

But,in extreme situation, if I did not start that master again, instead:
I shutdown the remaining master after inserting some documents, then,
I start the 2 masters, status goes green again, but the recently inserted documents won't be synced.

How can I get these documents synced?
I'll thanks for any ideas.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5931100E-5030-4AD7-82C2-0CE8E6016471%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Hi David, thanks a lot for the reply, and you are greatly correct.
I tested carefully (wait for master2 to be up and running before starting
master1), the document now synced.

But, I have a new question now, as real failover, when the 2 server maybe
goes down itself (not shutdonw manually),
I'll don't know which went down earlier or later, and this should be
automatic done, what can I do?

Also, note that in production, you should consider having 3 master nodes.

Thank for the hint, I'd like to know whether more node is necessary? for
example some data nodes (node.master: false; node.data: true)?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/07b5993c-04ef-4f25-a856-c67b0ed79783%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi David, thanks a lot for the reply, and you are greatly correct.
I tested carefully (wait for master2 to be up and running before starting
master1), the document now synced.

But, I have a new question now, as real failover, when the 2 server maybe
goes down itself (not shutdonw manually),
I'll don't know which went down earlier or later, and this should be
automatic done, what can I do?

Also, note that in production, you should consider having 3 master nodes.

Thank for the hint, I'd like to know whether more node is necessary? for
example some data nodes (node.master: false; node.data: true)?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Having 3 nodes makes sure you can easily maintain a majority quorum.
Once you get to larger sizes, it may/does make sense to have some data and
master only nodes.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 6 May 2014 13:02, victor fence fclddcn@gmail.com wrote:

Hi David, thanks a lot for the reply, and you are greatly correct.
I tested carefully (wait for master2 to be up and running before starting
master1), the document now synced.

But, I have a new question now, as real failover, when the 2 server maybe
goes down itself (not shutdonw manually),
I'll don't know which went down earlier or later, and this should be
automatic done, what can I do?

Also, note that in production, you should consider having 3 master nodes.

Thank for the hint, I'd like to know whether more node is necessary? for
example some data nodes (node.master: false; node.data: true)?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bN%3D%2B47MMY7z5U7P33FLYLX-aT2L_wHgkiT5%2BDYz%2BJFQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Having 3 nodes makes sure you can easily maintain a majority quorum.
Once you get to larger sizes, it may/does make sense to have some data
and master only nodes.

Thanks for your explanation, this is what I really need, thanks :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f458b3b7-b07c-4a69-a972-3cb37977de98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.