# Slow cluster startup with zen discovery and large number of nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2014, 2:40pm UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953 "2014-02-21T14:40:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [February 21, 2014, 2:40pm UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953/1 "2014-02-21T14:40:47Z")
Hi, Starting a cluster with 100 nodes takes half an hour just for the nodes to join in elasticsearch version 1.0. In version 0.19.8 nodes were very quick to join the cluster. The issue seems to come from the master node sending the updated state to all the nodes in the cluster after every single addition of a node and then waiting for the nodes to acknowledge the cluster update before adding the next node (zen-disco-receive). Setting discovery.zen.publish\_timeout:0 seems to resolve the issue during startup, because the master node does not block anymore, but I am not sure if something can go wrong afterwards while running the cluster with the timeout set to 0. I also tried setting increasing the kernel connections, but it did not make a difference: sysctl -w net.ipv4.tcp\_max\_syn\_backlog=20480 sysctl -w net.core.somaxconn=8192 sysctl -w net.ipv4.tcp\_syncookies=1 sysctl -w net.ipv4.tcp\_synack\_retries=1 So the question would be if it is safe to run the cluster with discovery.zen.publish\_timeout set to 0 and if the behavior is to be expected that zen discovery does not perform well for a larger number of nodes? Or if there might still be something wrong with the setup? Thanks in Advance, Michel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH0sEYisBxUpRksMbYox06sbONjtOPtRbc%2Btqzzg%2Bu5%3DVrbrcw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH0sEYisBxUpRksMbYox06sbONjtOPtRbc%2Btqzzg%2Bu5%3DVrbrcw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 21, 2014, 5:11pm UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953/2 "2014-02-21T17:11:43Z")
You may be interested in some settings that help a full cluster restart: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after) There is also a webinar that talks about some of the above: [http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/](http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Thibaut\_Britz\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibaut_britz_2/32/1222_2.png) [@Thibaut\_Britz\_2](https://discuss.elastic.co/u/Thibaut_Britz_2)\ **Post date:** [February 21, 2014, 5:44pm UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953/3 "2014-02-21T17:44:52Z")
Hi, I'm working with Michel on that issue: The cluster is completely empty and has no indexes at all. So it certainly is not related to revocery. The old elasticsearch version doesn't have code to wait for replies which causes the very slow startup. Thanks, Thibaut On Fri, Feb 21, 2014 at 6:11 PM, Binh Ly [binh@hibalo.com](mailto:binh@hibalo.com) wrote: > You may be interested in some settings that help a full cluster restart: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after) > > There is also a webinar that talks about some of the above: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicSiqGrCdfT-Xw5cAuuqLrct%3DsPb9Z\_JXB8EvvckonU54A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE_AicSiqGrCdfT-Xw5cAuuqLrct%3DsPb9Z_JXB8EvvckonU54A%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [February 24, 2014, 8:54am UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953/4 "2014-02-24T08:54:39Z")
Also the kernel complains about too many connections being made at once on the joining node. (Seems to occur after 30 nodes joined the cluster) TCP: TCP: Possible SYN flooding on port 9300. Sending cookies. Check SNMP counters. On Fri, Feb 21, 2014 at 6:44 PM, Thibaut Britz [t.britz@trendiction.com](mailto:t.britz@trendiction.com) wrote: > Hi, > > I'm working with Michel on that issue: > > The cluster is completely empty and has no indexes at all. So it certainly > is not related to revocery. > The old elasticsearch version doesn't have code to wait for replies which > causes the very slow startup. > > Thanks, > Thibaut > > On Fri, Feb 21, 2014 at 6:11 PM, Binh Ly [binh@hibalo.com](mailto:binh@hibalo.com) wrote: > > > You may be interested in some settings that help a full cluster restart: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after) > > > > There is also a webinar that talks about some of the above: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a405d32a-50a5-4d83-a3c7-8a0ea3449d28%40googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH0sEYjyTwFZFP7ZPCsOquSJDrYmdTrrhkmpD5XxbKpv-YO85A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH0sEYjyTwFZFP7ZPCsOquSJDrYmdTrrhkmpD5XxbKpv-YO85A%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:47am UTC](https://discuss.elastic.co/t/slow-cluster-startup-with-zen-discovery-and-large-number-of-nodes/15953/5 "2017-07-06T01:47:54Z")