# Ping timeouts, two elected master nodes, wrong metadata and deleted indices **URL:** \ **Category:** Elasticsearch\ **Created:** [January 30, 2012, 2:47pm UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539 "2012-01-30T14:47:38Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [January 30, 2012, 2:47pm UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/1 "2012-01-30T14:47:38Z")
Hi, I am getting ping timeouts from time to time, recently the master (192.168.5.4) did not reply fast enough (timeout is 1min), another master has been elected (192.168.5.2) and both stayed active. A status request on the old master gave a NullPointerException, a status request on the new master gave a green status with 39 nodes (the old master has been missing, and did not rejoin the cluster). A \_cluster/state request on the old master, gave as master still the old one, it did not lose its master status correctly, but the cluster as a whole seemed to be running, as all indices were green and the rivers were indexing correctly. After programmatically closing about 10 indices, the metadata was messed up. It seems that there are two master nodes working against each other. The old master updated his old version of the state and propageted the changes, resulting in an incorrect state with old information about shards, which were not on the indicated nodes anymore. It seems that to old and new master were on a race condition, about updating the index status, as afterwards some indices have been closed, while others remained open. The nodes were monitoring two masters, and accepted state from both, resulting in a restart of the fault detection. Afterwards the nodes started deleting the indices, which were not assigned on them at the point where the old master lost his master status. My current configuration is the following: es 0.17.10, 40 nodes, 5 master nodes, minimum master nodes set to 3. 300 indices and 30 rivers. shards per index: 1, replicas: 1. unicast discovery and local gateway. log extract from new masternode when the timeout came: (192.168.5.2) [2012-01-27 18:08:47,921][DEBUG][zen.fd][main] [Mondo] [master] uses ping\_interval [1s], ping\_timeout [1m], ping\_retries [3] [2012-01-27 18:08:47,922][DEBUG][zen.fd][main] [Mondo] [node] uses ping\_interval [1s], ping\_timeout [1m], ping\_retries [3] [2012-01-27 18:10:08,823][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-1] [Mondo] [master] starting fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [initial\_join] [2012-01-28 05:00:33,862][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-1047] [Mondo] [master] failed to ping [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], tried [3] times, each with maximum [1m] timeout [2012-01-28 05:00:33,863][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-1047] [Mondo] [master] stopping fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [master failure, failed to ping, tried [3] times, each with maximum [1m] timeout] [2012-01-28 05:00:33,863][INFO ][discovery.zen][elasticsearch[cached]-pool-11-thread-1044] [Mondo] master\_left [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [failed to ping, tried [3] times, each with maximum [1m] timeout] [2012-01-28 05:00:33,864][INFO ][cluster.service][elasticsearch[Mondo]clusterService#updateTask-pool-21-thread-1] [Mondo] master {new [Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}, previous [Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}}, removed {[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true},}, reason: zen-disco-master\_failed ([Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}) while closing the indices when the state got mixed up: (192.168.5.2) [2012-01-30 10:38:08,378][WARN][discovery.zen][New I/O server worker #1-7] [Mondo] master should not receive new cluster state from [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}] [2012-01-30 10:38:08,570][WARN][discovery.zen][New I/O server worker #1-7] [Mondo] master should not receive new cluster state from [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}] [2012-01-30 10:38:08,801][WARN][discovery.zen][New I/O server worker #1-7] [Mondo] master should not receive new cluster state from [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}] on the other nodes it seems that there are 2 master nodes: [2012-01-27 18:17:40,855][DEBUG][zen.fd][main] [Ev Teel Urizen] [master] uses ping\_interval [1s], ping\_timeout [1m], ping\_retries [3] [2012-01-27 18:17:40,857][DEBUG][zen.fd][main] [Ev Teel Urizen] [node ] uses ping\_interval [1s], ping\_timeout [1m], ping\_retries [3] [2012-01-27 18:18:01,728][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-1] [Ev Teel Urizen] [master] starting fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [initial\_join] [2012-01-27 18:18:07,766][INFO][discovery][main] [Ev Teel Urizen] trendictionsearch/a4rRT6\_BSNeceIpHf9AVkw [2012-01-28 05:06:52,403][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-832] [Ev Teel Urizen] [master] failed to ping [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], tried [3] times, each with maximum [1m] timeout [2012-01-28 05:06:52,404][DEBUG][zen.fd][elasticsearch[cached]-pool-11-thread-832] [Ev Teel Urizen] [master] stopping fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [master failure, failed to ping, tried [3] times, each with maximum [1m] timeout] [2012-01-28 05:06:52,406][DEBUG][zen.fd][elasticsearch[Ev Teel Urizen]clusterService#updateTask-pool-21-thread-1] [Ev Teel Urizen] [master] restarting fault detection against master [[Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}], reason [possible elected master since master left (reason = failed to ping, tried [3] times, each with maximum [1m] timeout)] [2012-01-28 09:26:56,244][DEBUG][zen.fd][elasticsearch[Ev Teel Urizen]clusterService#updateTask-pool-21-thread-1] [Ev Teel Urizen] [master] restarting fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [new cluster stare received and we monitor the wrong master [[Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}]] [2012-01-28 09:26:59,252][DEBUG][zen.fd][elasticsearch[Ev Teel Urizen]clusterService#updateTask-pool-21-thread-1] [Ev Teel Urizen] [master] restarting fault detection against master [[Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}], reason [new cluster stare received and we monitor the wrong master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}]] [2012-01-28 09:26:59,258][DEBUG][zen.fd][elasticsearch[Ev Teel Urizen]clusterService#updateTask-pool-21-thread-1] [Ev Teel Urizen] [master] restarting fault detection against master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}], reason [new cluster stare received and we monitor the wrong master [[Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}]] [2012-01-28 09:26:59,260][DEBUG][zen.fd][elasticsearch[Ev Teel Urizen]clusterService#updateTask-pool-21-thread-1] [Ev Teel Urizen] [master] restarting fault detection against master [[Mondo][BTiSBXfRQWqgTI1R19WGSA][inet[/192.168.5.2:9300]]{master=true}], reason [new cluster stare received and we monitor the wrong master [[Abner Little][-kmiCv3gTpqGqBmsgOBe0A][inet[/192.168.5.4:9300]]{master=true}]] afterwards on the datanodes themselves, data is being deleted (taking the wrong state into account): [2012-01-30 10:50:56,201][DEBUG][indices.store][elasticsearch[Gorilla Girl]clusterService#updateTask-pool-21-thread-1] [Gorilla Girl] [search\_index\_356] deleting index that is no longer in the cluster meta\_date Wanting to update to es 0.18, I am wondering if the same behaviour is still present in 0.18. The question here is that: - why does the old master not rejoin the cluster correctly (my best bet would be that he still things he is part of a cluster of which he still is the master, which would also explain why he updates the state). - why does the old master not lose its master status, when the other one is elected? Is the faultdetection behaving correctly when a node only temporarily goes down under load? - I thought that a node could only see one elected master node, and that the setting minimum\_master\_nodes set to 3 when there are 5 master nodes, would make it impossible for 2 master nodes to be active at the same time. - why are the timeouts on the ping arriving so regularly, can it be that it is some threading issue, like it was with the starting of the rivers after a full cluster restart, as we are using many indices, many nodes and many rivers. Also is it possible to not delete the indices locally on the nodes, so that I might have a chance to recover the deleted indices, by copying them over to the node were they are expected to be found. Thanks, Michel ---
**Author:** ![drew\_letcher](https://avatars.discourse-cdn.com/v4/letter/d/b3f665/32.png) [@drew\_letcher](https://discuss.elastic.co/u/drew_letcher)\ **Post date:** [January 31, 2012, 10:56pm UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/2 "2012-01-31T22:56:25Z")
We have seen a similar situation resulting in two masters using 0.18.6. We have 5 servers with master/data nodes and 10 servers with client nodes. Master was #5. I assume there was some type of network glitch, but we aren't sure exactly what happened. Then #2 took over as master and #1, #3, #4 and most of the client nodes (but not all) joined it. It seemed that #5 was still able to ping #1 and #4 so it thought it should stay master. Using the head plugin we could tell that #5 was trying to tell #1 and #4 to reallocate a shard. Though, since they were actually part of #2's cluster they seemed to ignore the command, because it stayed in that state until we bounced #5. I am guessing that the "ping" is not smart enough. It knows if another node is alive, but maybe it doesn't respond with any cluster info to know that the another node has claimed to be master??? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 1, 2012, 9:45am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/3 "2012-02-01T09:45:21Z")
Hi, Yea, this can happen. One way to help mitigate it is by setting the minimum\_master\_nodes in the configuration to a higher value (3 for a 5 node cluster, or 7 in a 10 node cluster), but it might still happen in complex disconnection scenarios (one way disconnect). One of the things left to add to zen discovery is automatic resolution of such scenarios, instead of having to go and bounce a specific node. -shay.banon On Wednesday, February 1, 2012 at 12:56 AM, drew.letcher wrote: > We have seen a similar situation resulting in two masters using > 0.18.6. We have 5 servers with master/data nodes and 10 servers with > client nodes. > > Master was #5. I assume there was some type of network glitch, but we > aren't sure exactly what happened. Then #2 took over as master and > #1, #3, #4 and most of the client nodes (but not all) joined it. > > It seemed that #5 was still able to ping #1 and #4 so it thought it > should stay master. Using the head plugin we could tell that #5 was > trying to tell #1 and #4 to reallocate a shard. Though, since they > were actually part of #2's cluster they seemed to ignore the command, > because it stayed in that state until we bounced #5. > > I am guessing that the "ping" is not smart enough. It knows if > another node is alive, but maybe it doesn't respond with any cluster > info to know that the another node has claimed to be master??? ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [February 1, 2012, 10:20am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/4 "2012-02-01T10:20:41Z")
Hi, the question is if there is a way to prevent the "wrong" master to send updates to the other nodes, resulting in an inconsistent state, finally leading to data loss, when the nodes are deleting data, that they think will not be used anymore. When we had the situation the nodes themselves switched masters between the correct new one and the old one (new cluster stare received and we monitor the wrong master). Afterwards while closing some indices, the old master received some of the updates, updated his wrong state and propagated the changes. So, yes the question would be if there is a possibility to detect and prevent a situation with two master nodes working against each other. Best, Michel On Wed, Feb 1, 2012 at 10:45 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Hi, > > Yea, this can happen. One way to help mitigate it is by setting the > minimum\_master\_nodes in the configuration to a higher value (3 for a 5 node > cluster, or 7 in a 10 node cluster), but it might still happen in complex > disconnection scenarios (one way disconnect). One of the things left to add > to zen discovery is automatic resolution of such scenarios, instead of > having to go and bounce a specific node. > > -shay.banon > > On Wednesday, February 1, 2012 at 12:56 AM, drew.letcher wrote: > > We have seen a similar situation resulting in two masters using > 0.18.6. We have 5 servers with master/data nodes and 10 servers with > client nodes. > > Master was #5. I assume there was some type of network glitch, but we > aren't sure exactly what happened. Then #2 took over as master and > #1, #3, #4 and most of the client nodes (but not all) joined it. > > It seemed that #5 was still able to ping #1 and #4 so it thought it > should stay master. Using the head plugin we could tell that #5 was > trying to tell #1 and #4 to reallocate a shard. Though, since they > were actually part of #2's cluster they seemed to ignore the command, > because it stayed in that state until we bounced #5. > > I am guessing that the "ping" is not smart enough. It knows if > another node is alive, but maybe it doesn't respond with any cluster > info to know that the another node has claimed to be master??? ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [February 1, 2012, 10:31am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/5 "2012-02-01T10:31:10Z")
Hi Shay, do you you recommend that we set minimum\_master\_nodes to (number of cluster nodes/2) in order to avoid this bug? Thanks, Thibaut On Wed, Feb 1, 2012 at 11:20 AM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > Hi, > the question is if there is a way to prevent the "wrong" master to > send updates to > the other nodes, resulting in an inconsistent state, finally leading > to data loss, when > the nodes are deleting data, that they think will not be used anymore. > > When we had the situation the nodes themselves switched masters between > the correct new one and the old one (new cluster stare received and we > monitor the wrong master). > Afterwards while closing some indices, the old master received some of > the updates, > updated his wrong state and propagated the changes. > > So, yes the question would be if there is a possibility to detect and > prevent a situation > with two master nodes working against each other. > > Best, > Michel > > On Wed, Feb 1, 2012 at 10:45 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Hi, > > > > Yea, this can happen. One way to help mitigate it is by setting the > > minimum\_master\_nodes in the configuration to a higher value (3 for a 5 > > node > > cluster, or 7 in a 10 node cluster), but it might still happen in complex > > disconnection scenarios (one way disconnect). One of the things left to > > add > > to zen discovery is automatic resolution of such scenarios, instead of > > having to go and bounce a specific node. > > > > -shay.banon > > > > On Wednesday, February 1, 2012 at 12:56 AM, drew.letcher wrote: > > > > We have seen a similar situation resulting in two masters using > > 0.18.6. We have 5 servers with master/data nodes and 10 servers with > > client nodes. > > > > Master was #5. I assume there was some type of network glitch, but we > > aren't sure exactly what happened. Then #2 took over as master and > > #1, #3, #4 and most of the client nodes (but not all) joined it. > > > > It seemed that #5 was still able to ping #1 and #4 so it thought it > > should stay master. Using the head plugin we could tell that #5 was > > trying to tell #1 and #4 to reallocate a shard. Though, since they > > were actually part of #2's cluster they seemed to ignore the command, > > because it stayed in that state until we bounced #5. > > > > I am guessing that the "ping" is not smart enough. It knows if > > another node is alive, but maybe it doesn't respond with any cluster > > info to know that the another node has claimed to be master??? ---
**Author:** ![drew\_letcher](https://avatars.discourse-cdn.com/v4/letter/d/b3f665/32.png) [@drew\_letcher](https://discuss.elastic.co/u/drew_letcher)\ **Post date:** [February 1, 2012, 10:34am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/6 "2012-02-01T10:34:32Z")
In my scenario node #1 and #4 should know that there are two masters for the same cluster name. They should force a re-election of a new master. This would minimize any loss of data, if it happens quick enough. Sounds like Shay is aware of the issue. We'll look forward to this enhancement in a future release. Thanks, Drew ---
**Author:** ![drew\_letcher](https://avatars.discourse-cdn.com/v4/letter/d/b3f665/32.png) [@drew\_letcher](https://discuss.elastic.co/u/drew_letcher)\ **Post date:** [February 1, 2012, 10:36am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/7 "2012-02-01T10:36:05Z")
BTW, our minimum\_master\_nodes is set to 3. ---
**Author:** ![drew\_letcher](https://avatars.discourse-cdn.com/v4/letter/d/b3f665/32.png) [@drew\_letcher](https://discuss.elastic.co/u/drew_letcher)\ **Post date:** [February 1, 2012, 11:04am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/8 "2012-02-01T11:04:35Z")
A couple more notes. We have redundant importers, but still lost data when this scenario occured. Luckily we planned for situations like this that cause gaps in the data. We used our recovery process to reindex the missing documents. Also, it took us a couple of hours to notice that node #5 had gone rogue. All others servers reported cluster health as green. We monitor the health of each node, but only monitor cluster health on one node. To detect this situation we should add monitors for cluster health on each node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:40am UTC](https://discuss.elastic.co/t/ping-timeouts-two-elected-master-nodes-wrong-metadata-and-deleted-indices/6539/9 "2017-07-06T03:40:51Z")