Nodes not joining after 1.4.0 upgrade

I've upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I'm able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn't want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don't want to join. :frowning: Thoughts?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasticsearch@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

But it DOES happen with 1.3.5. Hmmm….

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/369b9154662148c9b320d1b1fde952e5%40CY1PR0701MB1164.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

The logs you mentioned indicate that the nodes try to join the cluster but
it takes too long for a complete verification cycle (connect back to node
and publish cluster state to it) takes too long. It seems there is
something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast
discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s
much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0
after several hours of fiddling, but 1.4.0-beta did come up.

From: elasticsearch@googlegroups.com [
mailto:elasticsearch@googlegroups.com elasticsearch@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasticsearch@googlegroups.com [
mailto:elasticsearch@googlegroups.com elasticsearch@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasticsearch@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasticsearch@googlegroups.com [
mailto:elasticsearch@googlegroups.com elasticsearch@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasticsearch@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them,
I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3]
failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/522a6b85-394c-420f-8a99-6fb63998b532%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which was
bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:

Hi,

The logs you mentioned indicate that the nodes try to join the cluster but
it takes too long for a complete verification cycle (connect back to node
and publish cluster state to it) takes too long. It seems there is
something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast
discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com <javascript:> [mailto:
elasti...@googlegroups.com <javascript:>] *On Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s
much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0
after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com <javascript:>
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them,
I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi VincentYou should be able to do a rolling upgrade to version 1.4 from 1.3.4. Can you say more about the issues your were seeing? Any errors in the logs?

Cheers,

Boaz


Sent from Mailbox

On Mon, Nov 10, 2014 at 4:05 PM, Valentin pletzer@gmail.com wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:.....
I could get it up and running after restarting the whole cluster (which was
bad since I'm using it for realtime logging).
On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:

Hi,

The logs you mentioned indicate that the nodes try to join the cluster but
it takes too long for a complete verification cycle (connect back to node
and publish cluster state to it) takes too long. It seems there is
something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast
discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com <javascript:> [mailto:
elasti...@googlegroups.com <javascript:>] *On Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s
much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0
after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com <javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com <javascript:> [
mailto:ela...@googlegroups.com <javascript:>] *On Behalf Of *Janet
Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com <javascript:>
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them,
I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1415646940286.7852600e%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.

I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin <pletzer@gmail.commailto:pletzer@gmail.com> wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:
Hi,

The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:
More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com<javascript:> [mailto:elasti...@googlegroups.com<javascript:>] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com<javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com<javascript:> [mailto:ela...@googlegroups.com<javascript:>] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com<javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com<javascript:> [mailto:ela...@googlegroups.com<javascript:>] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com<javascript:>
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com<javascript:> [mailto:ela...@googlegroups.com<javascript:>] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com<javascript:>
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com<javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com<javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com<javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com<javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2A7A3BE8-1206-4B9E-BBAB-01AB5FFD2F36%40nairial.net.
For more options, visit https://groups.google.com/d/optout.

Hi Janet,

Was there anything in the master logs, i.e., gnslogstash10 in your
example?

Cheers,
Boaz

On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote:

I’m also using unicast discovery, as multicast doesn’t work on Azure. I
ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4
wouldn’t accept shards with the new lucene version. I ended up rebuilding
the cluster, and I’m going to have to backfill from text logs. A fresh 1.4
cluster works fine, but after two days I couldn’t get the upgraded cluster
to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin pletzer@gmail.com wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which
was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:

Hi,

The logs you mentioned indicate that the nodes try to join the cluster
but it takes too long for a complete verification cycle (connect back to
node and publish cluster state to it) takes too long. It seems there is
something going on your masters.

Can you check the logs over there? Also are you using multicast or
unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall
it’s much better on beta. I wasn’t able to get my 12 node cluster up on
1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of
them, I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

One more thing - do you use the Azure plugin for ES?

On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote:

Hi Janet,

Was there anything in the master logs, i.e., gnslogstash10 in your
example?

Cheers,
Boaz

On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote:

I’m also using unicast discovery, as multicast doesn’t work on Azure.
I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4
wouldn’t accept shards with the new lucene version. I ended up rebuilding
the cluster, and I’m going to have to backfill from text logs. A fresh 1.4
cluster works fine, but after two days I couldn’t get the upgraded cluster
to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin pletzer@gmail.com wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which
was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:

Hi,

The logs you mentioned indicate that the nodes try to join the cluster
but it takes too long for a complete verification cycle (connect back to
node and publish cluster state to it) takes too long. It seems there is
something going on your masters.

Can you check the logs over there? Also are you using multicast or
unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall
it’s much better on beta. I wasn’t able to get my 12 node cluster up on
1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of
them, I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master
[[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for
task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

There was nothing in the master logs. It’s like they were asleep.

From: Boaz Leskes <b.leskes@gmail.commailto:b.leskes@gmail.com>
Reply-To: "elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com" <elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com>
Date: Monday, November 10, 2014 at 10:50 PM
To: "elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com" <elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com>
Subject: Re: Nodes not joining after 1.4.0 upgrade

Hi Janet,

Was there anything in the master logs, i.e., gnslogstash10 in your example?

Cheers,
Boaz

On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote:
I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin <pletzer@gmail.commailto:pletzer@gmail.com> wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:
Hi,

The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:
More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D087022B.19053%janets%40nairial.net.
For more options, visit https://groups.google.com/d/optout.

No, I use unicast discovery, not the Azure plugin.

From: Boaz Leskes <b.leskes@gmail.commailto:b.leskes@gmail.com>
Reply-To: "elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com" <elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com>
Date: Monday, November 10, 2014 at 10:52 PM
To: "elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com" <elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com>
Subject: Re: Nodes not joining after 1.4.0 upgrade

One more thing - do you use the Azure plugin for ES?

On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote:
Hi Janet,

Was there anything in the master logs, i.e., gnslogstash10 in your example?

Cheers,
Boaz

On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote:
I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin <pletzer@gmail.commailto:pletzer@gmail.com> wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:
Hi,

The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters.

Can you check the logs over there? Also are you using multicast or unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:
More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] On Behalf Of Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:34:49,776][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:35:53,571][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:36:57,353][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:38:01,120][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:39:04,885][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2014-11-09 04:40:08,657][INFO ][discovery.zen ] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. :frowning: Thoughts?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%janets%40nairial.net.
For more options, visit https://groups.google.com/d/optout.

I see. Do you mind sharing the logs somewhere of the nodes? (privately via
e-mail also possible). I'd love to understand what happend.

On Tue Nov 11 2014 at 9:01:39 AM Janet Sullivan janets@nairial.net wrote:

No, I use unicast discovery, not the Azure plugin.

From: Boaz Leskes b.leskes@gmail.com
Reply-To: "elasticsearch@googlegroups.com" <elasticsearch@googlegroups.com

Date: Monday, November 10, 2014 at 10:52 PM
To: "elasticsearch@googlegroups.com" elasticsearch@googlegroups.com
Subject: Re: Nodes not joining after 1.4.0 upgrade

One more thing - do you use the Azure plugin for ES?

On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote:

Hi Janet,

Was there anything in the master logs, i.e., gnslogstash10 in your
example?

Cheers,
Boaz

On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote:

I’m also using unicast discovery, as multicast doesn’t work on Azure.
I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4
wouldn’t accept shards with the new lucene version. I ended up rebuilding
the cluster, and I’m going to have to backfill from text logs. A fresh 1.4
cluster works fine, but after two days I couldn’t get the upgraded cluster
to work. I’m glad to hear someone else had a similar issue.

On Nov 10, 2014, at 7:05 AM, Valentin pletzer@gmail.com wrote:

I had similar issues when upgrading from 1.3.4 to 1.4
from my elasticsearch.yml

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts:.....

I could get it up and running after restarting the whole cluster (which
was bad since I'm using it for realtime logging).

On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote:

Hi,

The logs you mentioned indicate that the nodes try to join the
cluster but it takes too long for a complete verification cycle (connect
back to node and publish cluster state to it) takes too long. It seems
there is something going on your masters.

Can you check the logs over there? Also are you using multicast or
unicast discovery?

On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote:

More hours of working – even when I get a 1.4.0 cluster up, masters
wouldn’t fail over – when I took master1 down, neither master2 or master3
would promote themselves. In 1.4.0-beta it fails over quickly.

From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com]
*On Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 11:11 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

OK, it also happens to some degree with 1.4.0-beta, although overall
it’s much better on beta. I wasn’t able to get my 12 node cluster up on
1.4.0 after several hours of fiddling, but 1.4.0-beta did come up.

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:26 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

But it DOES happen with 1.3.5. Hmmm….

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 9:24 PM
To: elasti...@googlegroups.com
Subject: RE: Nodes not joining after 1.4.0 upgrade

Note: This issue doesn’t happen with 1.4.0-beta1

From: elasti...@googlegroups.com [mailto:ela...@googlegroups.com] *On
Behalf Of *Janet Sullivan
Sent: Saturday, November 08, 2014 8:46 PM
To: elasti...@googlegroups.com
Subject: Nodes not joining after 1.4.0 upgrade

I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of
them, I had nodes that spewed the following, and were slow to join, if they
joined at all:

[2014-11-09 04:33:45,995][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:34:49,776][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:35:53,571][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:36:57,353][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:38:01,120][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:39:04,885][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

[2014-11-09 04:40:08,657][INFO ][discovery.zen ]
[gnslogstash3] failed to send join request to master [[gnslogstash10][9nx_f_
NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false,
master=true}], reason [ElasticsearchTimeoutException[Timeout waiting
for task.]]

I’m able to telnet to port 9300 on gnslogstash10 in this example from
gnslogstash3 with no issue, but this cluster doesn’t want to bring all its
nodes up. The more nodes added, the more likely a join will fail. In this
example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2817051de55a4f96a4bd173f26177f
29%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2098c745e47648d6b668b8c699f875
0c%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/47110bba4ae6400c9888beb70972c6
f0%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2a296bdb68984733af72024cfe8657
f6%40CY1PR0701MB1164.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%janets%40nairial.net
https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%janets%40nairial.net?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKzwz0qn3CbYzSA_gOCTnAOpFjVX%3Dt_CYGRQL6nFgveUfaVJDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.