# Nodes not joining after 1.4.0 upgrade **URL:** \ **Category:** Elasticsearch\ **Created:** [November 9, 2014, 4:45am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651 "2014-11-09T04:45:46Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 9, 2014, 4:45am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/1 "2014-11-09T04:45:46Z")
I've upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] I'm able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn't want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don't want to join. ☹ Thoughts? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 9, 2014, 5:23am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/2 "2014-11-09T05:23:59Z")
Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 9, 2014, 5:26am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/3 "2014-11-09T05:26:16Z")
But it DOES happen with 1.3.5. Hmmm…. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 9, 2014, 7:10am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/4 "2014-11-09T07:10:55Z")
OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:26 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade But it DOES happen with 1.3.5. Hmmm…. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 9, 2014, 7:35am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/5 "2014-11-09T07:35:49Z")
More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 11:11 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:26 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade But it DOES happen with 1.3.5. Hmmm…. From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/369b9154662148c9b320d1b1fde952e5%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/369b9154662148c9b320d1b1fde952e5%40CY1PR0701MB1164.namprd07.prod.outlook.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [November 10, 2014, 12:34pm UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/6 "2014-11-10T12:34:12Z")
Hi, The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters. Can you check the logs over there? Also are you using multicast or unicast discovery? On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > More hours of working – even when I get a 1.4.0 cluster up, masters > wouldn’t fail over – when I took master1 down, neither master2 or master3 > would promote themselves. In 1.4.0-beta it fails over quickly. > > _From:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [mailto: > [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] \*On Behalf Of \*Janet Sullivan > _Sent:_ Saturday, November 08, 2014 11:11 PM > _To:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > OK, it also happens to some degree with 1.4.0-beta, although overall it’s > much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 > after several hours of fiddling, but 1.4.0-beta did come up. > > _From:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [ > [mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] \*On > Behalf Of \*Janet Sullivan > _Sent:_ Saturday, November 08, 2014 9:26 PM > _To:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > But it DOES happen with 1.3.5. Hmmm…. > > _From:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [ > [mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] \*On > Behalf Of \*Janet Sullivan > _Sent:_ Saturday, November 08, 2014 9:24 PM > _To:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > Note: This issue doesn’t happen with 1.4.0-beta1 > > _From:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [ > [mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] \*On > Behalf Of \*Janet Sullivan > _Sent:_ Saturday, November 08, 2014 8:46 PM > _To:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > _Subject:_ Nodes not joining after 1.4.0 upgrade > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, > I had nodes that spewed the following, and were slow to join, if they > joined at all: > > [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] > failed to send join request to master > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > task.]] > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > nodes up. The more nodes added, the more likely a join will fail. In this > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com) > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com) > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com) > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com) > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/522a6b85-394c-420f-8a99-6fb63998b532%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/522a6b85-394c-420f-8a99-6fb63998b532%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Valentin\_Pletzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valentin_pletzer/32/105548_2.png) [@Valentin\_Pletzer](https://discuss.elastic.co/u/Valentin_Pletzer)\ **Post date:** [November 10, 2014, 3:05pm UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/7 "2014-11-10T15:05:19Z")
I had similar issues when upgrading from 1.3.4 to 1.4 from my elasticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts:..... I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging). On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: > Hi, > > The logs you mentioned indicate that the nodes try to join the cluster but > it takes too long for a complete verification cycle (connect back to node > and publish cluster state to it) takes too long. It seems there is > something going on your masters. > > Can you check the logs over there? Also are you using multicast or unicast > discovery? > > On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > > > More hours of working – even when I get a 1.4.0 cluster up, masters > > wouldn’t fail over – when I took master1 down, neither master2 or master3 > > would promote themselves. In 1.4.0-beta it fails over quickly. > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [mailto: > > [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \] \*On Behalf Of \*Janet Sullivan > > _Sent:_ Saturday, November 08, 2014 11:11 PM > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > OK, it also happens to some degree with 1.4.0-beta, although overall it’s > > much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 > > after several hours of fiddling, but 1.4.0-beta did come up. > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > Sullivan > > _Sent:_ Saturday, November 08, 2014 9:26 PM > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > But it DOES happen with 1.3.5. Hmmm…. > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > Sullivan > > _Sent:_ Saturday, November 08, 2014 9:24 PM > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > Note: This issue doesn’t happen with 1.4.0-beta1 > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > Sullivan > > _Sent:_ Saturday, November 08, 2014 8:46 PM > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > _Subject:_ Nodes not joining after 1.4.0 upgrade > > > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, > > I had nodes that spewed the following, and were slow to join, if they > > joined at all: > > > > [2014-11-09 04:33:45,995][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:34:49,776][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:35:53,571][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:36:57,353][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:38:01,120][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:39:04,885][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > [2014-11-09 04:40:08,657][INFO][discovery.zen] > > [gnslogstash3] failed to send join request to master > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > task.]] > > > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > > nodes up. The more nodes added, the more likely a join will fail. In this > > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [November 10, 2014, 7:15pm UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/8 "2014-11-10T19:15:41Z")
Hi VincentYou should be able to do a rolling upgrade to version 1.4 from 1.3.4. Can you say more about the issues your were seeing? Any errors in the logs? Cheers, Boaz — Sent from Mailbox On Mon, Nov 10, 2014 at 4:05 PM, Valentin [pletzer@gmail.com](mailto:pletzer@gmail.com) wrote: > I had similar issues when upgrading from 1.3.4 to 1.4 > from my elasticsearch.yml > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts:..... > I could get it up and running after restarting the whole cluster (which was > bad since I'm using it for realtime logging). > On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: > > > Hi, > > > > The logs you mentioned indicate that the nodes try to join the cluster but > > it takes too long for a complete verification cycle (connect back to node > > and publish cluster state to it) takes too long. It seems there is > > something going on your masters. > > > > Can you check the logs over there? Also are you using multicast or unicast > > discovery? > > > > On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > > > > > More hours of working – even when I get a 1.4.0 cluster up, masters > > > wouldn’t fail over – when I took master1 down, neither master2 or master3 > > > would promote themselves. In 1.4.0-beta it fails over quickly. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [mailto: > > > [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \] \*On Behalf Of \*Janet Sullivan > > > _Sent:_ Saturday, November 08, 2014 11:11 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > OK, it also happens to some degree with 1.4.0-beta, although overall it’s > > > much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 > > > after several hours of fiddling, but 1.4.0-beta did come up. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > > Sullivan > > > _Sent:_ Saturday, November 08, 2014 9:26 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > But it DOES happen with 1.3.5. Hmmm…. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > > Sullivan > > > _Sent:_ Saturday, November 08, 2014 9:24 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > Note: This issue doesn’t happen with 1.4.0-beta1 > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ [ > > > [mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com) \] \*On Behalf Of \*Janet > > > Sullivan > > > _Sent:_ Saturday, November 08, 2014 8:46 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \ > > > _Subject:_ Nodes not joining after 1.4.0 upgrade > > > > > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, > > > I had nodes that spewed the following, and were slow to join, if they > > > joined at all: > > > > > > [2014-11-09 04:33:45,995][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:34:49,776][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:35:53,571][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:36:57,353][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:38:01,120][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:39:04,885][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:40:08,657][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > > > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > > > nodes up. The more nodes added, the more likely a join will fail. In this > > > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1415646940286.7852600e%40Nodemailer](https://groups.google.com/d/msgid/elasticsearch/1415646940286.7852600e%40Nodemailer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 10, 2014, 10:47pm UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/9 "2014-11-10T22:47:12Z")
I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue. On Nov 10, 2014, at 7:05 AM, Valentin \<[pletzer@gmail.com](mailto:pletzer@gmail.com)[mailto:pletzer@gmail.com](mailto:pletzer@gmail.com)\> wrote: I had similar issues when upgrading from 1.3.4 to 1.4 from my elasticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts:..... I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging). On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: Hi, The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters. Can you check the logs over there? Also are you using multicast or unicast discovery? On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 11:11 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ Subject: RE: Nodes not joining after 1.4.0 upgrade OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)\] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:26 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ Subject: RE: Nodes not joining after 1.4.0 upgrade But it DOES happen with 1.3.5. Hmmm…. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)\] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)\] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)\ Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\. To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\. To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\. To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com)\. To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2A7A3BE8-1206-4B9E-BBAB-01AB5FFD2F36%40nairial.net](https://groups.google.com/d/msgid/elasticsearch/2A7A3BE8-1206-4B9E-BBAB-01AB5FFD2F36%40nairial.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [November 11, 2014, 6:50am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/10 "2014-11-11T06:50:46Z")
Hi Janet, Was there anything in the master logs, i.e., gnslogstash10 in your example? Cheers, Boaz On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: > I’m also using unicast discovery, as multicast doesn’t work on Azure. I > ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 > wouldn’t accept shards with the new lucene version. I ended up rebuilding > the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 > cluster works fine, but after two days I couldn’t get the upgraded cluster > to work. I’m glad to hear someone else had a similar issue. > > On Nov 10, 2014, at 7:05 AM, Valentin [pletzer@gmail.com](mailto:pletzer@gmail.com) wrote: > > I had similar issues when upgrading from 1.3.4 to 1.4 > from my elasticsearch.yml > > discovery.zen.ping.multicast.enabled: false > > discovery.zen.ping.unicast.hosts:..... > > I could get it up and running after restarting the whole cluster (which > was bad since I'm using it for realtime logging). > > On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: > > > Hi, > > > > The logs you mentioned indicate that the nodes try to join the cluster > > but it takes too long for a complete verification cycle (connect back to > > node and publish cluster state to it) takes too long. It seems there is > > something going on your masters. > > > > Can you check the logs over there? Also are you using multicast or > > unicast discovery? > > > > On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > > > > > More hours of working – even when I get a 1.4.0 cluster up, masters > > > wouldn’t fail over – when I took master1 down, neither master2 or master3 > > > would promote themselves. In 1.4.0-beta it fails over quickly. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)] \*On > > > Behalf Of \*Janet Sullivan > > > _Sent:_ Saturday, November 08, 2014 11:11 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > OK, it also happens to some degree with 1.4.0-beta, although overall > > > it’s much better on beta. I wasn’t able to get my 12 node cluster up on > > > 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > Behalf Of \*Janet Sullivan > > > _Sent:_ Saturday, November 08, 2014 9:26 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > But it DOES happen with 1.3.5. Hmmm…. > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > Behalf Of \*Janet Sullivan > > > _Sent:_ Saturday, November 08, 2014 9:24 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > Note: This issue doesn’t happen with 1.4.0-beta1 > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > Behalf Of \*Janet Sullivan > > > _Sent:_ Saturday, November 08, 2014 8:46 PM > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > _Subject:_ Nodes not joining after 1.4.0 upgrade > > > > > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of > > > them, I had nodes that spewed the following, and were slow to join, if they > > > joined at all: > > > > > > [2014-11-09 04:33:45,995][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:34:49,776][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:35:53,571][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:36:57,353][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:38:01,120][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:39:04,885][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > [2014-11-09 04:40:08,657][INFO][discovery.zen] > > > [gnslogstash3] failed to send join request to master > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > task.]] > > > > > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > > > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > > > nodes up. The more nodes added, the more likely a join will fail. In this > > > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [November 11, 2014, 6:52am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/11 "2014-11-11T06:52:02Z")
One more thing - do you use the Azure plugin for ES? On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote: > Hi Janet, > > Was there anything in the master logs, i.e., gnslogstash10 in your > example? > > Cheers, > Boaz > > On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: > > > I’m also using unicast discovery, as multicast doesn’t work on Azure. > > I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 > > wouldn’t accept shards with the new lucene version. I ended up rebuilding > > the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 > > cluster works fine, but after two days I couldn’t get the upgraded cluster > > to work. I’m glad to hear someone else had a similar issue. > > > > On Nov 10, 2014, at 7:05 AM, Valentin [pletzer@gmail.com](mailto:pletzer@gmail.com) wrote: > > > > I had similar issues when upgrading from 1.3.4 to 1.4 > > from my elasticsearch.yml > > > > discovery.zen.ping.multicast.enabled: false > > > > discovery.zen.ping.unicast.hosts:..... > > > > I could get it up and running after restarting the whole cluster (which > > was bad since I'm using it for realtime logging). > > > > On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: > > > > > Hi, > > > > > > The logs you mentioned indicate that the nodes try to join the cluster > > > but it takes too long for a complete verification cycle (connect back to > > > node and publish cluster state to it) takes too long. It seems there is > > > something going on your masters. > > > > > > Can you check the logs over there? Also are you using multicast or > > > unicast discovery? > > > > > > On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > > > > > > > More hours of working – even when I get a 1.4.0 cluster up, masters > > > > wouldn’t fail over – when I took master1 down, neither master2 or master3 > > > > would promote themselves. In 1.4.0-beta it fails over quickly. > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)] \*On > > > > Behalf Of \*Janet Sullivan > > > > _Sent:_ Saturday, November 08, 2014 11:11 PM > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > OK, it also happens to some degree with 1.4.0-beta, although overall > > > > it’s much better on beta. I wasn’t able to get my 12 node cluster up on > > > > 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > Behalf Of \*Janet Sullivan > > > > _Sent:_ Saturday, November 08, 2014 9:26 PM > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > But it DOES happen with 1.3.5. Hmmm…. > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > Behalf Of \*Janet Sullivan > > > > _Sent:_ Saturday, November 08, 2014 9:24 PM > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > Note: This issue doesn’t happen with 1.4.0-beta1 > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > Behalf Of \*Janet Sullivan > > > > _Sent:_ Saturday, November 08, 2014 8:46 PM > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > _Subject:_ Nodes not joining after 1.4.0 upgrade > > > > > > > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of > > > > them, I had nodes that spewed the following, and were slow to join, if they > > > > joined at all: > > > > > > > > [2014-11-09 04:33:45,995][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:34:49,776][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:35:53,571][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:36:57,353][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:38:01,120][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:39:04,885][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > [2014-11-09 04:40:08,657][INFO][discovery.zen] > > > > [gnslogstash3] failed to send join request to master > > > > [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for > > > > task.]] > > > > > > > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > > > > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > > > > nodes up. The more nodes added, the more likely a join will fail. In this > > > > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 11, 2014, 8:01am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/12 "2014-11-11T08:01:09Z")
There was nothing in the master logs. It’s like they were asleep. From: Boaz Leskes \<[b.leskes@gmail.com](mailto:b.leskes@gmail.com)[mailto:b.leskes@gmail.com](mailto:b.leskes@gmail.com)\> Reply-To: "elasticsearch@googlegroups.com[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" \<[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)\> Date: Monday, November 10, 2014 at 10:50 PM To: "elasticsearch@googlegroups.com[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" \<[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)\> Subject: Re: Nodes not joining after 1.4.0 upgrade Hi Janet, Was there anything in the master logs, i.e., gnslogstash10 in your example? Cheers, Boaz On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue. On Nov 10, 2014, at 7:05 AM, Valentin \<[pletzer@gmail.com](mailto:pletzer@gmail.com)[mailto:pletzer@gmail.com](mailto:pletzer@gmail.com)\> wrote: I had similar issues when upgrading from 1.3.4 to 1.4 from my elasticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts:..... I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging). On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: Hi, The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters. Can you check the logs over there? Also are you using multicast or unicast discovery? On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 11:11 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:26 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade But it DOES happen with 1.3.5. Hmmm…. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4ddc8151-5c10-4e35-871d-655316ef2d48%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/D087022B.19053%janets%40nairial.net](https://groups.google.com/d/msgid/elasticsearch/D087022B.19053%25janets%40nairial.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Janet](https://avatars.discourse-cdn.com/v4/letter/j/5e9695/32.png) [@Janet](https://discuss.elastic.co/u/Janet)\ **Post date:** [November 11, 2014, 8:01am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/13 "2014-11-11T08:01:30Z")
No, I use unicast discovery, not the Azure plugin. From: Boaz Leskes \<[b.leskes@gmail.com](mailto:b.leskes@gmail.com)[mailto:b.leskes@gmail.com](mailto:b.leskes@gmail.com)\> Reply-To: "elasticsearch@googlegroups.com[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" \<[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)\> Date: Monday, November 10, 2014 at 10:52 PM To: "elasticsearch@googlegroups.com[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" \<[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)\> Subject: Re: Nodes not joining after 1.4.0 upgrade One more thing - do you use the Azure plugin for ES? On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote: Hi Janet, Was there anything in the master logs, i.e., gnslogstash10 in your example? Cheers, Boaz On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: I’m also using unicast discovery, as multicast doesn’t work on Azure. I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 wouldn’t accept shards with the new lucene version. I ended up rebuilding the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 cluster works fine, but after two days I couldn’t get the upgraded cluster to work. I’m glad to hear someone else had a similar issue. On Nov 10, 2014, at 7:05 AM, Valentin \<[pletzer@gmail.com](mailto:pletzer@gmail.com)[mailto:pletzer@gmail.com](mailto:pletzer@gmail.com)\> wrote: I had similar issues when upgrading from 1.3.4 to 1.4 from my elasticsearch.yml discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts:..... I could get it up and running after restarting the whole cluster (which was bad since I'm using it for realtime logging). On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: Hi, The logs you mentioned indicate that the nodes try to join the cluster but it takes too long for a complete verification cycle (connect back to node and publish cluster state to it) takes too long. It seems there is something going on your masters. Can you check the logs over there? Also are you using multicast or unicast discovery? On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: More hours of working – even when I get a 1.4.0 cluster up, masters wouldn’t fail over – when I took master1 down, neither master2 or master3 would promote themselves. In 1.4.0-beta it fails over quickly. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 11:11 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade OK, it also happens to some degree with 1.4.0-beta, although overall it’s much better on beta. I wasn’t able to get my 12 node cluster up on 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:26 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade But it DOES happen with 1.3.5. Hmmm…. From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 9:24 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: RE: Nodes not joining after 1.4.0 upgrade Note: This issue doesn’t happen with 1.4.0-beta1 From: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] On Behalf Of Janet Sullivan Sent: Saturday, November 08, 2014 8:46 PM To: [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) Subject: Nodes not joining after 1.4.0 upgrade I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of them, I had nodes that spewed the following, and were slow to join, if they joined at all: [2014-11-09 04:33:45,995][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:34:49,776][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:35:53,571][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:36:57,353][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:38:01,120][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:39:04,885][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] [2014-11-09 04:40:08,657][INFO][discovery.zen] [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]] ## I’m able to telnet to port 9300 on gnslogstash10 in this example from gnslogstash3 with no issue, but this cluster doesn’t want to bring all its nodes up. The more nodes added, the more likely a join will fail. In this example, 9 nodes are up, but 3 nodes don’t want to join. ☹ Thoughts? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com)[https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%janets%40nairial.net](https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%25janets%40nairial.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [November 11, 2014, 8:48am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/14 "2014-11-11T08:48:07Z")
I see. Do you mind sharing the logs somewhere of the nodes? (privately via e-mail also possible). I'd love to understand what happend. On Tue Nov 11 2014 at 9:01:39 AM Janet Sullivan [janets@nairial.net](mailto:janets@nairial.net) wrote: > No, I use unicast discovery, not the Azure plugin. > > From: Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) > Reply-To: "[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" \<[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > > > > > Date: Monday, November 10, 2014 at 10:52 PM > To: "[elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)" [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > Subject: Re: Nodes not joining after 1.4.0 upgrade > > One more thing - do you use the Azure plugin for ES? > > On Tuesday, November 11, 2014 7:50:47 AM UTC+1, Boaz Leskes wrote: > > > Hi Janet, > > > > Was there anything in the master logs, i.e., gnslogstash10 in your > > example? > > > > Cheers, > > Boaz > > > > On Monday, November 10, 2014 11:47:22 PM UTC+1, Janet Sullivan wrote: > > > > > I’m also using unicast discovery, as multicast doesn’t work on Azure. > > > I ended up in a bad position - 1.4 wouldn’t come up all the way, but 1.3.4 > > > wouldn’t accept shards with the new lucene version. I ended up rebuilding > > > the cluster, and I’m going to have to backfill from text logs. A fresh 1.4 > > > cluster works fine, but after two days I couldn’t get the upgraded cluster > > > to work. I’m glad to hear someone else had a similar issue. > > > > > > On Nov 10, 2014, at 7:05 AM, Valentin [pletzer@gmail.com](mailto:pletzer@gmail.com) wrote: > > > > > > I had similar issues when upgrading from 1.3.4 to 1.4 > > > from my elasticsearch.yml > > > > > > discovery.zen.ping.multicast.enabled: false > > > > > > discovery.zen.ping.unicast.hosts:..... > > > > > > I could get it up and running after restarting the whole cluster (which > > > was bad since I'm using it for realtime logging). > > > > > > On Monday, November 10, 2014 1:34:12 PM UTC+1, Boaz Leskes wrote: > > > > > > > Hi, > > > > > > > > The logs you mentioned indicate that the nodes try to join the > > > > cluster but it takes too long for a complete verification cycle (connect > > > > back to node and publish cluster state to it) takes too long. It seems > > > > there is something going on your masters. > > > > > > > > Can you check the logs over there? Also are you using multicast or > > > > unicast discovery? > > > > > > > > On Sunday, November 9, 2014 8:36:06 AM UTC+1, Janet Sullivan wrote: > > > > > > > > > More hours of working – even when I get a 1.4.0 cluster up, masters > > > > > wouldn’t fail over – when I took master1 down, neither master2 or master3 > > > > > would promote themselves. In 1.4.0-beta it fails over quickly. > > > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:elasti...@googlegroups.com](mailto:elasti...@googlegroups.com)] > > > > > \*On Behalf Of \*Janet Sullivan > > > > > _Sent:_ Saturday, November 08, 2014 11:11 PM > > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > > > OK, it also happens to some degree with 1.4.0-beta, although overall > > > > > it’s much better on beta. I wasn’t able to get my 12 node cluster up on > > > > > 1.4.0 after several hours of fiddling, but 1.4.0-beta did come up. > > > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > > Behalf Of \*Janet Sullivan > > > > > _Sent:_ Saturday, November 08, 2014 9:26 PM > > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > > > But it DOES happen with 1.3.5. Hmmm…. > > > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > > Behalf Of \*Janet Sullivan > > > > > _Sent:_ Saturday, November 08, 2014 9:24 PM > > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > > _Subject:_ RE: Nodes not joining after 1.4.0 upgrade > > > > > > > > > > Note: This issue doesn’t happen with 1.4.0-beta1 > > > > > > > > > > _From:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) [[mailto:ela...@googlegroups.com](mailto:ela...@googlegroups.com)] \*On > > > > > Behalf Of \*Janet Sullivan > > > > > _Sent:_ Saturday, November 08, 2014 8:46 PM > > > > > _To:_ [elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) > > > > > _Subject:_ Nodes not joining after 1.4.0 upgrade > > > > > > > > > > I’ve upgraded a couple of clusters to 1.4.0 from 1.3.4. On both of > > > > > them, I had nodes that spewed the following, and were slow to join, if they > > > > > joined at all: > > > > > > > > > > [2014-11-09 04:33:45,995][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:34:49,776][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:35:53,571][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:36:57,353][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:38:01,120][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:39:04,885][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > [2014-11-09 04:40:08,657][INFO][discovery.zen] > > > > > [gnslogstash3] failed to send join request to master [[gnslogstash10][9nx\_f\_ > > > > > NiQtCntgnD2q7k0g][gnslogstash10][inet[/10.0.0.29:9300]]{data=false, > > > > > master=true}], reason [ElasticsearchTimeoutException[Timeout waiting > > > > > for task.]] > > > > > > > > > > I’m able to telnet to port 9300 on gnslogstash10 in this example from > > > > > gnslogstash3 with no issue, but this cluster doesn’t want to bring all its > > > > > nodes up. The more nodes added, the more likely a join will fail. In this > > > > > example, 9 nodes are up, but 3 nodes don’t want to join. L Thoughts? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/2817051de55a4f96a4bd173f26177f > > > > > 29%[40CY1PR0701MB1164.namprd07.prod.outlook.com](http://40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2817051de55a4f96a4bd173f26177f29%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/2098c745e47648d6b668b8c699f875 > > > > > 0c%[40CY1PR0701MB1164.namprd07.prod.outlook.com](http://40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2098c745e47648d6b668b8c699f8750c%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/47110bba4ae6400c9888beb70972c6 > > > > > f0%[40CY1PR0701MB1164.namprd07.prod.outlook.com](http://40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/47110bba4ae6400c9888beb70972c6f0%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/2a296bdb68984733af72024cfe8657 > > > > > f6%[40CY1PR0701MB1164.namprd07.prod.outlook.com](http://40CY1PR0701MB1164.namprd07.prod.outlook.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2a296bdb68984733af72024cfe8657f6%40CY1PR0701MB1164.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02354045-84fb-4820-a9ea-8b06279cead9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > ``` > > > -- > > > > > > ``` > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/86bb1cde-be20-43bc-a832-8364603174cc%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/lD7vn56Y7sM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%25janets%40nairial.net](https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%25janets%40nairial.net) > [https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%25janets%40nairial.net?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/D087024A.19055%25janets%40nairial.net?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKzwz0qn3CbYzSA\_gOCTnAOpFjVX%3Dt\_CYGRQL6nFgveUfaVJDw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKzwz0qn3CbYzSA_gOCTnAOpFjVX%3Dt_CYGRQL6nFgveUfaVJDw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:50am UTC](https://discuss.elastic.co/t/nodes-not-joining-after-1-4-0-upgrade/20651/15 "2017-07-06T00:50:52Z")