# Problem of sync elasticsearch data after failover **URL:** \ **Category:** Elasticsearch\ **Created:** [May 4, 2014, 1:50am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333 "2014-05-04T01:50:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![fclddcn](https://avatars.discourse-cdn.com/v4/letter/f/278dde/32.png) [@fclddcn](https://discuss.elastic.co/u/fclddcn)\ **Post date:** [May 4, 2014, 1:50am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/1 "2014-05-04T01:50:18Z")
Hi everyone, I am new to elasticsearch. I'v setted up a small cluster (with version 0.90.1). Within the cluster I have 4 nodes total: loadblancer node 1: on app1 server loadblancer node 2: on app2 server master node 1: stand alone server 1 master node 2: stand alone server 2 This architecture is come from the idea in this discussion: > See Duc.Duong's reply each master node have config as: node.master: true node.data: true discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [":9300",":9300"] I can get cluster have green status now. For me, the most important thing is failover, and now this is basicly avaliable. That is to say, I can shut down one master (using command "service elasticsearch stop") and ES still work correctly, and when I start that master again, it joins the cluster as a new candidate master, the status goes green, and data gets synced. But,in extreme situation, if I did not start that master again, instead: I shutdown the remaining master after inserting some documents, then, I start the 2 masters, status goes green again, but the recently inserted documents won't be synced. How can I get these documents synced? I'll thanks for any ideas. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 4, 2014, 7:14am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/2 "2014-05-04T07:14:00Z")
I think you restarted master1 first (the one who have less documents) and then master2. Try to do the opposite and wait for master2 to be up and running before starting master1. Also, note that in production, you should consider having 3 master nodes. My 2 cents -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 4 mai 2014 à 03:50, cl f [fclddcn@gmail.com](mailto:fclddcn@gmail.com) a écrit : Hi everyone, I am new to elasticsearch. I'v setted up a small cluster (with version 0.90.1). Within the cluster I have 4 nodes total: loadblancer node 1: on app1 server loadblancer node 2: on app2 server master node 1: stand alone server 1 master node 2: stand alone server 2 This architecture is come from the idea in this discussion: > See Duc.Duong's reply each master node have config as: node.master: true node.data: true discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [":9300",":9300"] I can get cluster have green status now. For me, the most important thing is failover, and now this is basicly avaliable. That is to say, I can shut down one master (using command "service elasticsearch stop") and ES still work correctly, and when I start that master again, it joins the cluster as a new candidate master, the status goes green, and data gets synced. But,in extreme situation, if I did not start that master again, instead: I shutdown the remaining master after inserting some documents, then, I start the 2 masters, status goes green again, but the recently inserted documents won't be synced. ## How can I get these documents synced? I'll thanks for any ideas. You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/91c325aa-59f7-4527-9ed2-fef6dbe44733%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5931100E-5030-4AD7-82C2-0CE8E6016471%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/5931100E-5030-4AD7-82C2-0CE8E6016471%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fclddcn](https://avatars.discourse-cdn.com/v4/letter/f/278dde/32.png) [@fclddcn](https://discuss.elastic.co/u/fclddcn)\ **Post date:** [May 4, 2014, 8:56am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/3 "2014-05-04T08:56:35Z")
Hi David, thanks a lot for the reply, and you are greatly correct. I tested carefully (wait for master2 to be up and running before starting master1), the document now synced. But, I have a new question now, as real failover, when the 2 server maybe goes down itself (not shutdonw manually), I'll don't know which went down earlier or later, and this should be automatic done, what can I do? > Also, note that in production, you should consider having 3 master nodes. Thank for the hint, I'd like to know whether more node is necessary? for example some data nodes (node.master: false; node.data: true)? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/07b5993c-04ef-4f25-a856-c67b0ed79783%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/07b5993c-04ef-4f25-a856-c67b0ed79783%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fclddcn](https://avatars.discourse-cdn.com/v4/letter/f/278dde/32.png) [@fclddcn](https://discuss.elastic.co/u/fclddcn)\ **Post date:** [May 6, 2014, 3:02am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/4 "2014-05-06T03:02:27Z")
Hi David, thanks a lot for the reply, and you are greatly correct. I tested carefully (wait for master2 to be up and running before starting master1), the document now synced. But, I have a new question now, as real failover, when the 2 server maybe goes down itself (not shutdonw manually), I'll don't know which went down earlier or later, and this should be automatic done, what can I do? > Also, note that in production, you should consider having 3 master nodes. Thank for the hint, I'd like to know whether more node is necessary? for example some data nodes (node.master: false; node.data: true)? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 6, 2014, 3:22am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/5 "2014-05-06T03:22:28Z")
Having 3 nodes makes sure you can easily maintain a majority quorum. Once you get to larger sizes, it may/does make sense to have some data and master only nodes. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 6 May 2014 13:02, victor fence [fclddcn@gmail.com](mailto:fclddcn@gmail.com) wrote: > Hi David, thanks a lot for the reply, and you are greatly correct. > I tested carefully (wait for master2 to be up and running before starting > master1), the document now synced. > > But, I have a new question now, as real failover, when the 2 server maybe > goes down itself (not shutdonw manually), > I'll don't know which went down earlier or later, and this should be > automatic done, what can I do? > > > Also, note that in production, you should consider having 3 master nodes. > > Thank for the hint, I'd like to know whether more node is necessary? for > example some data nodes (node.master: false; node.data: true)? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/320c7587-9f1f-464d-90db-f258e14cc533%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bN%3D%2B47MMY7z5U7P33FLYLX-aT2L\_wHgkiT5%2BDYz%2BJFQg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bN%3D%2B47MMY7z5U7P33FLYLX-aT2L_wHgkiT5%2BDYz%2BJFQg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fclddcn](https://avatars.discourse-cdn.com/v4/letter/f/278dde/32.png) [@fclddcn](https://discuss.elastic.co/u/fclddcn)\ **Post date:** [May 7, 2014, 1:46am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/6 "2014-05-07T01:46:19Z")
> Having 3 nodes makes sure you can easily maintain a majority quorum. > Once you get to larger sizes, it may/does make sense to have some data > and master only nodes. Thanks for your explanation, this is what I really need, thanks 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f458b3b7-b07c-4a69-a972-3cb37977de98%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f458b3b7-b07c-4a69-a972-3cb37977de98%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:31am UTC](https://discuss.elastic.co/t/problem-of-sync-elasticsearch-data-after-failover/17333/7 "2017-07-06T01:31:13Z")