Hi,
I'm using client nodes in my Java app and have 70+ of them (one in each indexing process).
(I have 10+ data nodes in addition to these client nodes.)
I sometimes see flood of multicast discovery pings in ES log (I turned on discovery TRACE),
and ES seems to become too busy responding pings and looses master.
I have two questions:
(1) It seems like one of the client nodes starts the ping flood. Is there a way to know what happened?
(2) Is there a way to prevent the flood?
Here's an example log at one ES node (original master was [Tenpin]):
[2012-09-29 16:39:47,000][TRACE][discovery.zen.ping.multicast] [Aero] [6] received ping_request from [[Dreadnought][40yJCCGDQIuf7LIcuXU28g][inet[/10.5.128.221:9309]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:47,009][TRACE][discovery.zen.ping.multicast] [Aero] [6] received ping_request from [[Llyra][rOZCc3YTSwWOEgtZXp-p5w][inet[/10.5.124.236:9300]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
< hundreds of these>
[2012-09-29 16:39:49,675][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], retry [3] out of [3]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [Tenpin][inet[/10.5.128.104:9300]][discovery/zen/fd/masterPing] request_id [9570831] timed out after [120002ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:347)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
[2012-09-29 16:39:49,675][DEBUG][discovery.zen.fd ] [Aero] [master] failed to ping [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], tried [3] times, each with maximum [2m] timeout
[2012-09-29 16:39:49,694][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], reason [master failure, failed to ping, tried [3] times, each with maximum [2m] timeout]
[2012-09-29 16:39:49,695][INFO ][discovery.zen ] [Aero] master_left [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], reason [failed to ping, tried [3] times, each with maximum [2m] timeout]
[2012-09-29 16:39:49,697][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]]], reason [possible elected master since master left (reason = failed to ping, tried [3] times, each with maximum [2m] timeout)]
[2012-09-29 16:39:49,697][INFO ][cluster.service ] [Aero] master {new [Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]], previous [Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]}, removed {[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]],}, reason: zen-disco-master_failed ([Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]])
[2012-09-29 16:39:50,760][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]] that is no longer a master
[2012-09-29 16:39:50,760][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]]], reason [master failure, no longer master]
[2012-09-29 16:39:50,760][INFO ][discovery.zen ] [Aero] master_left [[Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]]], reason [no longer master]
[2012-09-29 16:39:50,760][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Black Bolt][inet[/10.5.122.53:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:39:52,581][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], reason [possible elected master since master left (reason = no longer master)]
[2012-09-29 16:39:52,582][INFO ][cluster.service ] [Aero] master {new [Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]], previous [Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]]}, removed {[Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]],}, reason: zen-disco-master_failed ([Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]])
[2012-09-29 16:39:53,585][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]] that is no longer a master
[2012-09-29 16:39:53,585][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], reason [master failure, no longer master]
[2012-09-29 16:39:53,586][INFO ][discovery.zen ] [Aero] master_left [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], reason [no longer master]
[2012-09-29 16:39:53,586][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Zarrko, the Tomorrow Man][inet[/10.5.120.89:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:39:54,282][TRACE][discovery.zen.ping.multicast] [Aero] [7] received ping_request from [[Man-Brute][a3R-E0w9TZiFSTZpZ59yBQ][inet[/10.5.128.224:9310]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:54,289][TRACE][discovery.zen.ping.multicast] [Aero] [7] received ping_request from [[Magus][EnuJGEzxQsKshsopicOuZA][inet[/10.5.128.223:9301]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:54,343][TRACE][discovery.zen.ping.multicast] [Aero] [7] received ping_request from [[Unicorn][-VBiBf34SJCrHfvZqoEOjQ][inet[/10.5.124.236:9307]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:55,639][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]], reason [possible elected master since master left (reason = no longer master)]
[2012-09-29 16:39:55,639][INFO ][cluster.service ] [Aero] master {new [Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]], previous [Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]}, removed {[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]],}, reason: zen-disco-master_failed ([Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]])
[2012-09-29 16:39:56,642][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]] that is no longer a master
[2012-09-29 16:39:56,643][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]], reason [master failure, no longer master]
[2012-09-29 16:39:56,643][INFO ][discovery.zen ] [Aero] master_left [[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]], reason [no longer master]
[2012-09-29 16:39:56,643][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Sinclair, Rahne][inet[/10.5.120.87:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:39:58,584][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Purge][5I-HNNYwSoadnZ5ek_XWSQ][inet[/10.5.128.223:9303]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:58,588][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Ripfire][1CslAqxMRMSt8NGusEZwWA][inet[/10.5.128.221:9312]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:58,600][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]], reason [possible elected master since master left (reason = no longer master)]
[2012-09-29 16:39:58,600][INFO ][cluster.service ] [Aero] master {new [Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]], previous [Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]]}, removed {[Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]],}, reason: zen-disco-master_failed ([Sinclair, Rahne][5yucFAQTSYqZDLnlSUp46A][inet[/10.5.120.87:9300]])
[2012-09-29 16:39:58,646][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Agent Zero][R457XfoyS6emfsVnRHHeQQ][inet[/10.5.128.224:9301]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:58,663][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Eric the Red][0mmOX6B3QiiQCuxSYv8IKA][inet[/10.5.128.222:9311]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:39:59,604][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]] that is no longer a master
[2012-09-29 16:39:59,604][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]], reason [master failure, no longer master]
[2012-09-29 16:39:59,604][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Balor][inet[/10.5.124.110:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:39:59,604][INFO ][discovery.zen ] [Aero] master_left [[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]], reason [no longer master]
[2012-09-29 16:40:01,380][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], reason [possible elected master since master left (reason = no longer master)]
[2012-09-29 16:40:01,380][INFO ][cluster.service ] [Aero] master {new [Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]], previous [Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]]}, removed {[Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]],}, reason: zen-disco-master_failed ([Balor][9T3oEWEIQ-qp_pdUQ19qcw][inet[/10.5.124.110:9300]])
[2012-09-29 16:40:01,571][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Hitman][Y8IXpV-lStC6QcEg8qcDWQ][inet[/10.5.128.222:9304]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:01,584][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[Lorvex][GdM5ZsPwTbOaOl_wO_n0sQ][inet[/10.5.128.221:9311]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:01,608][TRACE][discovery.zen.ping.multicast] [Aero] [4] received ping_request from [[En Sabah Nur][Vh5kvLYiRdGAlRK7mfocPg][inet[/10.5.128.223:9311]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:02,383][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]] that is no longer a master
[2012-09-29 16:40:02,384][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], reason [master failure, no longer master]
[2012-09-29 16:40:02,384][INFO ][discovery.zen ] [Aero] master_left [[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]], reason [no longer master]
[2012-09-29 16:40:02,384][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Beautiful Dreamer][inet[/10.5.120.63:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:40:04,216][DEBUG][discovery.zen.fd ] [Aero] [master] restarting fault detection against master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], reason [possible elected master since master left (reason = no longer master)]
[2012-09-29 16:40:04,217][INFO ][cluster.service ] [Aero] master {new [Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]], previous [Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]]}, removed {[Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]],}, reason: zen-disco-master_failed ([Beautiful Dreamer][KIRKCmkSR5qqGFBG2ZsPPQ][inet[/10.5.120.63:9300]])
[2012-09-29 16:40:05,221][DEBUG][discovery.zen.fd ] [Aero] [master] pinging a master [Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]] that is no longer a master
[2012-09-29 16:40:05,221][DEBUG][discovery.zen.fd ] [Aero] [master] stopping fault detection against master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], reason [master failure, no longer master]
[2012-09-29 16:40:05,221][INFO ][discovery.zen ] [Aero] master_left [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], reason [no longer master]
[2012-09-29 16:40:05,221][TRACE][discovery.zen.fd ] [Aero] [master] failed to ping [null], retry [1] out of [3]
org.elasticsearch.transport.RemoteTransportException: [Terminatrix][inet[/10.5.120.90:9300]][discovery/zen/fd/masterPing]
Caused by: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$NoLongerMasterException
[2012-09-29 16:40:06,647][TRACE][discovery.zen.ping.multicast] [Aero] [5] received ping_request from [[Spoor][4FC_TBNfQ_WZgoGgjIWeNg][inet[/10.5.128.225:9303]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:06,716][TRACE][discovery.zen.ping.multicast] [Aero] [5] received ping_request from [[Madame Menace][0tM63hSDSGmrxlK247Qe7g][inet[/10.5.128.221:9310]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:06,747][TRACE][discovery.zen.ping.multicast] [Aero] [5] received ping_request from [[Nuklo][sktgA6c-T3GL6xhDWN-M6g][inet[/10.5.128.221:9302]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,079][TRACE][discovery.zen.ping.multicast] [Aero] [5] received ping_request from [[U-Man][oXjZ-YJbQEOoqKaNsB10jw][inet[/10.5.128.221:9306]]{client=true, data=false}], sending ping_response{target [[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]]], master [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,091][WARN ][discovery.zen ] [Aero] not enough master nodes after master left (reason = no longer master), current nodes: {[Dreadnought][40yJCCGDQIuf7LIcuXU28g][inet[/10.5.128.221:9309]]{client=true, data=false},[Night Rider][DBFmQH6aQoWGhc-epUbCXw][inet[/10.5.128.222:9310]]{client=true, data=false},[Hangman][_1MVh01bQxaQepfJaDIXMA][inet[/10.5.124.236:9311]]{client=true, data=false},[Torso][jamfX273QT--sLCYFVhBHw][inet[/10.5.124.236:9310]]{client=true, data=false},[Bandit][rTc74yegRQyLURgGOxuWdg][inet[/10.5.124.236:9309]]{client=true, data=false},[Turner D. Century][BLAqaK_3QhCoRFJOIlNL1g][inet[/10.5.128.224:9307]]{client=true, data=false},[Unseen][tkPuDGBeT_uzgvZALUA-ag][inet[/10.5.128.225:9309]]{client=true, data=false},[Unicorn][-VBiBf34SJCrHfvZqoEOjQ][inet[/10.5.124.236:9307]]{client=true, data=false},[Black Marvel][91D3roH0RR-CENza0HIjyw][inet[/10.5.128.224:9304]]{client=true, data=false},[Stuntmaster][A6v4PtcjQy-2u9yKkhO-sw][inet[/10.5.128.223:9307]]{client=true, data=false},[Burner][Tas26q5IRS66zqWhU1HXQQ][inet[/10.5.128.223:9309]]{client=true, data=false},[Riot Grrl][eQOzZhbST1afGhhizyn60g][inet[/10.5.124.236:9302]]{client=true, data=false},[Lorvex][GdM5ZsPwTbOaOl_wO_n0sQ][inet[/10.5.128.221:9311]]{client=true, data=false},[Saint Elmo][m2oziO9dTOuTcsQV0cLimQ][inet[/10.5.128.225:9306]]{client=true, data=false},[Stallior][SawpDECfQxGOyiVpvEL01Q][inet[/10.5.122.52:9300]],[Frost, Cordelia][czPHWqYmRT2bJD5cLw8r5g][inet[/10.5.128.224:9303]]{client=true, data=false},[Alpha the Ultimate Mutant][ef1tG851Sw-g15X1ZcbuRw][inet[/10.5.128.225:9304]]{client=true, data=false},[Amphibian][cLnG4tYhR_W8cafNGtwy1A][inet[/10.5.128.221:9305]]{client=true, data=false},[Cecilia Reyes][kIcAYwd7ToCRvebdmWoMEA][inet[/10.5.128.225:9307]]{client=true, data=false},[Eric the Red][0mmOX6B3QiiQCuxSYv8IKA][inet[/10.5.128.222:9311]]{client=true, data=false},[Venomm][MVxb4a_RQday3Kzu8jzQcg][inet[/10.5.128.224:9302]]{client=true, data=false},[Tigra][012bJL3RRsuHoTvhWTsmSA][inet[/10.5.128.225:9301]]{client=true, data=false},[Solo][7QBZccJDTN6ALsrbZQhOjw][inet[/10.5.128.222:9303]]{client=true, data=false},[Agent Zero][R457XfoyS6emfsVnRHHeQQ][inet[/10.5.128.224:9301]]{client=true, data=false},[Blaze][kXxzX1erQLqXDo_pU5OOiQ][inet[/10.5.128.221:9303]]{client=true, data=false},[Fly][fpAhLLo_RjWL9De7Qcai_w][inet[/10.5.128.223:9308]]{client=true, data=false},[Master Order][nzik66PkTZSi0RhleWv5Rg][inet[/10.5.128.225:9311]]{client=true, data=false},[Lorvex][vCN446CRRNmFctrKMMlqMQ][inet[/10.5.128.222:9306]]{client=true, data=false},[Dormammu][yg6JgSUwRXKwn94hzbxIcA][inet[/10.5.128.223:9300]]{client=true, data=false},[Spoor][4FC_TBNfQ_WZgoGgjIWeNg][inet[/10.5.128.225:9303]]{client=true, data=false},[Mimic][eyZY2KBzTbCcuY9iuKJ1GQ][inet[/10.5.124.236:9301]]{client=true, data=false},[Purge][5I-HNNYwSoadnZ5ek_XWSQ][inet[/10.5.128.223:9303]]{client=true, data=false},[Paste-Pot Pete][TaggV58SQdOBdorEX_7Cag][inet[/10.5.128.225:9302]]{client=true, data=false},[Madame Menace][0tM63hSDSGmrxlK247Qe7g][inet[/10.5.128.221:9310]]{client=true, data=false},[Agent Zero][jGDJ7WPCQD-0Lu9trfJDLQ][inet[/10.5.128.222:9302]]{client=true, data=false},[Spellbinder][ST4RXsIbTa6w2I2WZTIJHA][inet[/10.5.128.222:9309]]{client=true, data=false},[Gideon, Gregory][XzzIyGItRUCI4wcvcCg4hg][inet[/10.5.128.224:9305]]{client=true, data=false},[Puck][2r5UAyn5QHiiOZK7jo1HCg][inet[/10.5.124.236:9303]]{client=true, data=false},[Satana][6VIHTZwxTi2OKrkAeXjATQ][inet[/10.5.128.224:9306]]{client=true, data=false},[Callisto][k-4Be4RCSMan6ht3Tb1Dxg][inet[/10.5.128.221:9308]]{client=true, data=false},[Terraxia][YpwwD4axTTKYG_J5QDy_ig][inet[/10.5.128.223:9305]]{client=true, data=false},[En Sabah Nur][Vh5kvLYiRdGAlRK7mfocPg][inet[/10.5.128.223:9311]]{client=true, data=false},[Aero][kxWhZM7TQu2paHopxaBGIQ][inet[/10.5.120.86:9300]],[Raye, Frankie][xO0uJ4UJRceYSufA3aJCmQ][inet[/10.5.128.223:9304]]{client=true, data=false},[Bulldozer][tvCoj920Riq_0ISQ7xFLjA][inet[/10.5.128.224:9300]]{client=true, data=false},[Nital, Taj][EXkH8cToSuWmxTlP6BKHBw][inet[/10.5.128.221:9307]]{client=true, data=false},[Man-Brute][a3R-E0w9TZiFSTZpZ59yBQ][inet[/10.5.128.224:9310]]{client=true, data=false},[Dragonfly][iHFRlDCnQ5-VewjMDMye3g][inet[/10.5.124.236:9304]]{client=true, data=false},[Major Mapleleaf][FXMQWk8BRO2lhbYl2Xc6qA][inet[/10.5.128.222:9308]]{client=true, data=false},[Anything][v36j8MZzSxazEFh1wUS14g][inet[/10.5.128.222:9305]]{client=true, data=false},[O'Hara, Miguel][o8K1QaqATmqWZGcHjneKiQ][inet[/10.5.128.105:9300]],[Kamo Tharnn][eZxZYOGOSeaqRbEUf5RkSw][inet[/10.5.128.223:9306]]{client=true, data=false},[Llyra][rOZCc3YTSwWOEgtZXp-p5w][inet[/10.5.124.236:9300]]{client=true, data=false},[Capricorn][QkMi2IGzSUWUK8Ig5bm1Pg][inet[/10.5.128.224:9308]]{client=true, data=false},[Locksmith][4fa6OfghS86PCcZmKFc0tg][inet[/10.5.128.224:9309]]{client=true, data=false},[Professor Power][v4tRpe4vTAuIIsLbqiwy4w][inet[/10.5.128.221:9304]]{client=true, data=false},[Hawkshaw][OadjwK-CQ76mRDNT2q18kQ][inet[/10.5.128.221:9300]]{client=true, data=false},[Hitman][Y8IXpV-lStC6QcEg8qcDWQ][inet[/10.5.128.222:9304]]{client=true, data=false},[Titanium Man][mfaFyT9lSmKVuy1_HaP81A][inet[/10.5.120.88:9300]],[Sleepwalker][hRPxqiWrRkOrs7c8MKEwsg][inet[/10.5.128.222:9307]]{client=true, data=false},[Hawkeye][wTc_zMcVSDKCp9pz02cXWA][inet[/10.5.124.236:9305]]{client=true, data=false},[Catiana][P-g8xnCVT4qENrWSBvQ0CQ][inet[/10.5.128.222:9300]]{client=true, data=false},[Magus][EnuJGEzxQsKshsopicOuZA][inet[/10.5.128.223:9301]]{client=true, data=false},[U-Man][oXjZ-YJbQEOoqKaNsB10jw][inet[/10.5.128.221:9306]]{client=true, data=false},[Patch][YSaqvr5iTi6iW-hA5Yya3A][inet[/10.5.120.65:9300]],[Postman][8l5f580GTfinJ50xw7tHFA][inet[/10.5.124.236:9306]]{client=true, data=false},[Anti-Cap][RvVB-x2XQm2u2BINxLYFKg][inet[/10.5.128.225:9308]]{client=true, data=false},[Tuc][sKtEQl7BQZep8boAc-NFBw][inet[/10.5.124.236:9312]]{client=true, data=false},[Nuklo][sktgA6c-T3GL6xhDWN-M6g][inet[/10.5.128.221:9302]]{client=true, data=false},[Aged Genghis][ULU6rv9gQwyLONWLl8E7AA][inet[/10.5.128.225:9305]]{client=true, data=false},[Random][SMel_DgXTjalcIaDm2rW0w][inet[/10.5.128.225:9310]]{client=true, data=false},[Reaper][b_UQ-e7lSxCBCsQSctUAXQ][inet[/10.5.120.85:9300]],[Hercules][HcaoofIuTm-xq5BpZJxUjw][inet[/10.5.128.223:9302]]{client=true, data=false},[Ripfire][1CslAqxMRMSt8NGusEZwWA][inet[/10.5.128.221:9312]]{client=true, data=false},[War][OSmLDwLqQCa4SifhfLgVKw][inet[/10.5.124.236:9308]]{client=true, data=false},[Stephen Colbert][nYS0DbNDQFiMUeFGsEGowg][inet[/10.5.128.221:9301]]{client=true, data=false},[Armor][ZYDhNqDWToy_y-R3QpP8EQ][inet[/10.5.128.222:9301]]{client=true, data=false},[Skull the Slayer][2Wep7Vm-RtmNvLwF5dXjDQ][inet[/10.5.128.223:9310]]{client=true, data=false},[Thumbelina][0XXw7lYgS-67CQFYi4vGag][inet[/10.5.128.225:9300]]{client=true, data=false},}
[2012-09-29 16:40:07,092][TRACE][discovery.zen.ping.multicast] [Aero] [3] sending ping request
[2012-09-29 16:40:07,092][TRACE][discovery.zen.ping.multicast] [Aero] [3] received ping_response{target [[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]]], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,101][TRACE][discovery.zen.ping.multicast] [Aero] [3] received ping_response{target [[Zarrko, the Tomorrow Man][5lY5iLNYRvS-4mV_EJNOcQ][inet[/10.5.120.89:9300]]], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,101][TRACE][discovery.zen.ping.multicast] [Aero] [3] received ping_response{target [[Black Bolt][5k4qTPOxQMm9p2JGXckIOQ][inet[/10.5.122.53:9300]]], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,101][TRACE][discovery.zen.ping.multicast] [Aero] [3] received ping_response{target [[Agent Zero][jGDJ7WPCQD-0Lu9trfJDLQ][inet[/10.5.128.222:9302]]{client=true, data=false}], master [[Tenpin][05xjDBKOSxWMS0o2fi5IdQ][inet[/10.5.128.104:9300]]], cluster_name[es-cluster1]}
[2012-09-29 16:40:07,101][INFO ][cluster.service ] [Aero] removed {[Dreadnought][40yJCCGDQIuf7LIcuXU28g][inet[/10.5.128.221:9309]]{client=true, data=false},[Night Rider][DBFmQH6aQoWGhc-epUbCXw][inet[/10.5.128.222:9310]]{client=true, data=false},[Hangman][_1MVh01bQxaQepfJaDIXMA][inet[/10.5.124.236:9311]]{client=true, data=false},[Torso][jamfX273QT--sLCYFVhBHw][inet[/10.5.124.236:9310]]{client=true, data=false},[Bandit][rTc74yegRQyLURgGOxuWdg][inet[/10.5.124.236:9309]]{client=true, data=false},[Turner D. Century][BLAqaK_3QhCoRFJOIlNL1g][inet[/10.5.128.224:9307]]{client=true, data=false},[Unseen][tkPuDGBeT_uzgvZALUA-ag][inet[/10.5.128.225:9309]]{client=true, data=false},[Unicorn][-VBiBf34SJCrHfvZqoEOjQ][inet[/10.5.124.236:9307]]{client=true, data=false},[Black Marvel][91D3roH0RR-CENza0HIjyw][inet[/10.5.128.224:9304]]{client=true, data=false},[Stuntmaster][A6v4PtcjQy-2u9yKkhO-sw][inet[/10.5.128.223:9307]]{client=true, data=false},[Burner][Tas26q5IRS66zqWhU1HXQQ][inet[/10.5.128.223:9309]]{client=true, data=false},[Riot Grrl][eQOzZhbST1afGhhizyn60g][inet[/10.5.124.236:9302]]{client=true, data=false},[Lorvex][GdM5ZsPwTbOaOl_wO_n0sQ][inet[/10.5.128.221:9311]]{client=true, data=false},[Frost, Cordelia][czPHWqYmRT2bJD5cLw8r5g][inet[/10.5.128.224:9303]]{client=true, data=false},[Stallior][SawpDECfQxGOyiVpvEL01Q][inet[/10.5.122.52:9300]],[Saint Elmo][m2oziO9dTOuTcsQV0cLimQ][inet[/10.5.128.225:9306]]{client=true, data=false},[Alpha the Ultimate Mutant][ef1tG851Sw-g15X1ZcbuRw][inet[/10.5.128.225:9304]]{client=true, data=false},[Amphibian][cLnG4tYhR_W8cafNGtwy1A][inet[/10.5.128.221:9305]]{client=true, data=false},[Cecilia Reyes][kIcAYwd7ToCRvebdmWoMEA][inet[/10.5.128.225:9307]]{client=true, data=false},[Eric the Red][0mmOX6B3QiiQCuxSYv8IKA][inet[/10.5.128.222:9311]]{client=true, data=false},[Venomm][MVxb4a_RQday3Kzu8jzQcg][inet[/10.5.128.224:9302]]{client=true, data=false},[Tigra][012bJL3RRsuHoTvhWTsmSA][inet[/10.5.128.225:9301]]{client=true, data=false},[Solo][7QBZccJDTN6ALsrbZQhOjw][inet[/10.5.128.222:9303]]{client=true, data=false},[Blaze][kXxzX1erQLqXDo_pU5OOiQ][inet[/10.5.128.221:9303]]{client=true, data=false},[Agent Zero][R457XfoyS6emfsVnRHHeQQ][inet[/10.5.128.224:9301]]{client=true, data=false},[Fly][fpAhLLo_RjWL9De7Qcai_w][inet[/10.5.128.223:9308]]{client=true, data=false},[Master Order][nzik66PkTZSi0RhleWv5Rg][inet[/10.5.128.225:9311]]{client=true, data=false},[Lorvex][vCN446CRRNmFctrKMMlqMQ][inet[/10.5.128.222:9306]]{client=true, data=false},[Dormammu][yg6JgSUwRXKwn94hzbxIcA][inet[/10.5.128.223:9300]]{client=true, data=false},[Spoor][4FC_TBNfQ_WZgoGgjIWeNg][inet[/10.5.128.225:9303]]{client=true, data=false},[Mimic][eyZY2KBzTbCcuY9iuKJ1GQ][inet[/10.5.124.236:9301]]{client=true, data=false},[Purge][5I-HNNYwSoadnZ5ek_XWSQ][inet[/10.5.128.223:9303]]{client=true, data=false},[Paste-Pot Pete][TaggV58SQdOBdorEX_7Cag][inet[/10.5.128.225:9302]]{client=true, data=false},[Madame Menace][0tM63hSDSGmrxlK247Qe7g][inet[/10.5.128.221:9310]]{client=true, data=false},[Agent Zero][jGDJ7WPCQD-0Lu9trfJDLQ][inet[/10.5.128.222:9302]]{client=true, data=false},[Spellbinder][ST4RXsIbTa6w2I2WZTIJHA][inet[/10.5.128.222:9309]]{client=true, data=false},[Gideon, Gregory][XzzIyGItRUCI4wcvcCg4hg][inet[/10.5.128.224:9305]]{client=true, data=false},[Puck][2r5UAyn5QHiiOZK7jo1HCg][inet[/10.5.124.236:9303]]{client=true, data=false},[Satana][6VIHTZwxTi2OKrkAeXjATQ][inet[/10.5.128.224:9306]]{client=true, data=false},[Callisto][k-4Be4RCSMan6ht3Tb1Dxg][inet[/10.5.128.221:9308]]{client=true, data=false},[Terraxia][YpwwD4axTTKYG_J5QDy_ig][inet[/10.5.128.223:9305]]{client=true, data=false},[En Sabah Nur][Vh5kvLYiRdGAlRK7mfocPg][inet[/10.5.128.223:9311]]{client=true, data=false},[Raye, Frankie][xO0uJ4UJRceYSufA3aJCmQ][inet[/10.5.128.223:9304]]{client=true, data=false},[Bulldozer][tvCoj920Riq_0ISQ7xFLjA][inet[/10.5.128.224:9300]]{client=true, data=false},[Nital, Taj][EXkH8cToSuWmxTlP6BKHBw][inet[/10.5.128.221:9307]]{client=true, data=false},[Man-Brute][a3R-E0w9TZiFSTZpZ59yBQ][inet[/10.5.128.224:9310]]{client=true, data=false},[Dragonfly][iHFRlDCnQ5-VewjMDMye3g][inet[/10.5.124.236:9304]]{client=true, data=false},[Major Mapleleaf][FXMQWk8BRO2lhbYl2Xc6qA][inet[/10.5.128.222:9308]]{client=true, data=false},[Anything][v36j8MZzSxazEFh1wUS14g][inet[/10.5.128.222:9305]]{client=true, data=false},[Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]],[O'Hara, Miguel][o8K1QaqATmqWZGcHjneKiQ][inet[/10.5.128.105:9300]],[Llyra][rOZCc3YTSwWOEgtZXp-p5w][inet[/10.5.124.236:9300]]{client=true, data=false},[Kamo Tharnn][eZxZYOGOSeaqRbEUf5RkSw][inet[/10.5.128.223:9306]]{client=true, data=false},[Capricorn][QkMi2IGzSUWUK8Ig5bm1Pg][inet[/10.5.128.224:9308]]{client=true, data=false},[Locksmith][4fa6OfghS86PCcZmKFc0tg][inet[/10.5.128.224:9309]]{client=true, data=false},[Hawkshaw][OadjwK-CQ76mRDNT2q18kQ][inet[/10.5.128.221:9300]]{client=true, data=false},[Professor Power][v4tRpe4vTAuIIsLbqiwy4w][inet[/10.5.128.221:9304]]{client=true, data=false},[Hitman][Y8IXpV-lStC6QcEg8qcDWQ][inet[/10.5.128.222:9304]]{client=true, data=false},[Titanium Man][mfaFyT9lSmKVuy1_HaP81A][inet[/10.5.120.88:9300]],[Sleepwalker][hRPxqiWrRkOrs7c8MKEwsg][inet[/10.5.128.222:9307]]{client=true, data=false},[Hawkeye][wTc_zMcVSDKCp9pz02cXWA][inet[/10.5.124.236:9305]]{client=true, data=false},[Catiana][P-g8xnCVT4qENrWSBvQ0CQ][inet[/10.5.128.222:9300]]{client=true, data=false},[Magus][EnuJGEzxQsKshsopicOuZA][inet[/10.5.128.223:9301]]{client=true, data=false},[Patch][YSaqvr5iTi6iW-hA5Yya3A][inet[/10.5.120.65:9300]],[U-Man][oXjZ-YJbQEOoqKaNsB10jw][inet[/10.5.128.221:9306]]{client=true, data=false},[Postman][8l5f580GTfinJ50xw7tHFA][inet[/10.5.124.236:9306]]{client=true, data=false},[Anti-Cap][RvVB-x2XQm2u2BINxLYFKg][inet[/10.5.128.225:9308]]{client=true, data=false},[Tuc][sKtEQl7BQZep8boAc-NFBw][inet[/10.5.124.236:9312]]{client=true, data=false},[Aged Genghis][ULU6rv9gQwyLONWLl8E7AA][inet[/10.5.128.225:9305]]{client=true, data=false},[Nuklo][sktgA6c-T3GL6xhDWN-M6g][inet[/10.5.128.221:9302]]{client=true, data=false},[Random][SMel_DgXTjalcIaDm2rW0w][inet[/10.5.128.225:9310]]{client=true, data=false},[Reaper][b_UQ-e7lSxCBCsQSctUAXQ][inet[/10.5.120.85:9300]],[Ripfire][1CslAqxMRMSt8NGusEZwWA][inet[/10.5.128.221:9312]]{client=true, data=false},[Hercules][HcaoofIuTm-xq5BpZJxUjw][inet[/10.5.128.223:9302]]{client=true, data=false},[War][OSmLDwLqQCa4SifhfLgVKw][inet[/10.5.124.236:9308]]{client=true, data=false},[Stephen Colbert][nYS0DbNDQFiMUeFGsEGowg][inet[/10.5.128.221:9301]]{client=true, data=false},[Skull the Slayer][2Wep7Vm-RtmNvLwF5dXjDQ][inet[/10.5.128.223:9310]]{client=true, data=false},[Armor][ZYDhNqDWToy_y-R3QpP8EQ][inet[/10.5.128.222:9301]]{client=true, data=false},[Thumbelina][0XXw7lYgS-67CQFYi4vGag][inet[/10.5.128.225:9300]]{client=true, data=false},}, reason: zen-disco-master_failed ([Terminatrix][Os6cwU4IT9qWcGGpBVdZWg][inet[/10.5.120.90:9300]])
Thank you for your help.