# Max latency between nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [April 21, 2014, 8:58pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115 "2014-04-21T20:58:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![skik2skis](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@skik2skis](https://discuss.elastic.co/u/skik2skis)\ **Post date:** [April 21, 2014, 8:58pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/1 "2014-04-21T20:58:29Z")
I currently have an elasticsearch cluster with 7 nodes. Some of the connectivity between nodes is across fiber with 2 - 3ms latency between nodes. About once a day we see a node drop from the cluster, a new master is elected, and then the dropped node returns to the cluster usually 30 - 45 seconds later. The configuration on all nodes has been tweaked as follows to help tolerate the slight increase in latency but still seems to get a timeout when they drop. Is it expected that even 2ms of latency would cause issues with the cluster? If so, is there further configuration needed to make the cluster more tolerant of the latency? Or should this latency be expected and I should investigate other root causes for the nodes dropping occasionally? I've confirmed that we're never actually dropping packets between nodes, so something is going on that is causing them to not respond 5x60s pings. zen-disco-node\_failed([CDPX-PRD-ELS4][lkquUBfHT1aXAO3-\_tCNCg][cdpx-prd-els4][inet[10.9.64.142/10.9.64.142:9300]]{master=false}), reason failed to ping, tried [5] times, each with maximum [1m] timeout discovery.zen.fd.ping\_interval: 15s discovery.zen.fd.ping\_timeout: 60s discovery.zen.fd.ping\_retries: 5 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd49b82c-5496-48fd-8c8b-c47a42bb6d21%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd49b82c-5496-48fd-8c8b-c47a42bb6d21%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [April 21, 2014, 9:40pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/2 "2014-04-21T21:40:59Z")
Other than network, is it possible that your nodes could sometimes be overloaded such that they cannot respond immediately? If that's the case, then you can probably get 3 nodes (servers), make them master-only nodes (node.master: true, node.data: false). Set discovery.zen.minimum\_master\_nodes: 2 for those 3 nodes. And then for the rest of your other data nodes, make them non-master eligible (node.master: false, node.data: true). This way you have 3 nodes dedicated only to do cluster state/master tasks unimpeded by load or anything else other than your network. Just don't run anything else on them or send queries/indexing jobs to these 3 nodes. 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d8ad1d6d-2aaf-47ef-a901-4f661311dc25%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d8ad1d6d-2aaf-47ef-a901-4f661311dc25%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![skik2skis](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@skik2skis](https://discuss.elastic.co/u/skik2skis)\ **Post date:** [April 21, 2014, 9:51pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/3 "2014-04-21T21:51:51Z")
We currently are running dedicated master nodes but I believe they are also servicing queries. I can change it such that queries only hit the data nodes and see if that eliminates the issue... On Monday, April 21, 2014 3:40:59 PM UTC-6, Binh Ly wrote: > Other than network, is it possible that your nodes could sometimes be > overloaded such that they cannot respond immediately? If that's the case, > then you can probably get 3 nodes (servers), make them master-only nodes > (node.master: true, node.data: false). Set > discovery.zen.minimum\_master\_nodes: 2 for those 3 nodes. And then for the > rest of your other data nodes, make them non-master eligible (node.master: > false, node.data: true). This way you have 3 nodes dedicated only to do > cluster state/master tasks unimpeded by load or anything else other than > your network. Just don't run anything else on them or send queries/indexing > jobs to these 3 nodes. 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 25, 2014, 11:49pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/4 "2014-04-25T23:49:44Z")
Hey, is there any reason in the logfile of the master node, why it was deelected? (network outage as well)? Did you give your master nodes also a huge heap which could cause long outages during GC? --Alex On Mon, Apr 21, 2014 at 5:51 PM, [skik2skis@gmail.com](mailto:skik2skis@gmail.com) wrote: > We currently are running dedicated master nodes but I believe they are > also servicing queries. I can change it such that queries only hit the > data nodes and see if that eliminates the issue... > > On Monday, April 21, 2014 3:40:59 PM UTC-6, Binh Ly wrote: > > > Other than network, is it possible that your nodes could sometimes be > > overloaded such that they cannot respond immediately? If that's the case, > > then you can probably get 3 nodes (servers), make them master-only nodes > > (node.master: true, node.data: false). Set discovery.zen.minimum\_master\_nodes: > > 2 for those 3 nodes. And then for the rest of your other data nodes, make > > them non-master eligible (node.master: false, node.data: true). This way > > you have 3 nodes dedicated only to do cluster state/master tasks unimpeded > > by load or anything else other than your network. Just don't run anything > > else on them or send queries/indexing jobs to these 3 nodes. 🙂 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8uaMs3GANv-eF2TKbbfmOCjDuyVj-kEQ%3Dxi9AdZ8vsQw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8uaMs3GANv-eF2TKbbfmOCjDuyVj-kEQ%3Dxi9AdZ8vsQw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![skik2skis](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@skik2skis](https://discuss.elastic.co/u/skik2skis)\ **Post date:** [April 28, 2014, 3:39pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/5 "2014-04-28T15:39:04Z")
So far the only log message we've seen is: zen-disco-node\_failed([CDPX-PRD-ELS4][lkquUBfHT1aXAO3-\_tCNCg][cdpx-prd-els4][inet[ 10.9.64.142/10.9.64.142:9300]]{master=false}[http://10.9.64.142/10.9.64.142:9300]]{master=false}](http://10.9.64.142/10.9.64.142:9300%5D%5D%7Bmaster=false%7D)), reason failed to ping, tried [5] times, each with maximum [1m] timeout We have other data traversing the network that would be very sensitive to any latency or outages, in addition to alerts that would fire off if we had a network outage, so I am confident we don't have any network issues when this occurs. Furthermore, we are only seeing data nodes drop, the masters never drop. Is there a recommended heap size for nodes that are masters only? In addition, any recommendations on heap size for data nodes? I assume this could be a timeout in general during GC processes as our data nodes have larger heaps? On Friday, April 25, 2014 5:49:44 PM UTC-6, Alexander Reelsen wrote: > Hey, > > is there any reason in the logfile of the master node, why it was > deelected? (network outage as well)? Did you give your master nodes also a > huge heap which could cause long outages during GC? > > --Alex > > On Mon, Apr 21, 2014 at 5:51 PM, \<[skik...@gmail.com](mailto:skik...@gmail.com) \\> wrote: > > > We currently are running dedicated master nodes but I believe they are > > also servicing queries. I can change it such that queries only hit the > > data nodes and see if that eliminates the issue... > > > > On Monday, April 21, 2014 3:40:59 PM UTC-6, Binh Ly wrote: > > > > > Other than network, is it possible that your nodes could sometimes be > > > overloaded such that they cannot respond immediately? If that's the case, > > > then you can probably get 3 nodes (servers), make them master-only nodes > > > (node.master: true, node.data: false). Set discovery.zen.minimum\_master\_nodes: > > > 2 for those 3 nodes. And then for the rest of your other data nodes, make > > > them non-master eligible (node.master: false, node.data: true). This way > > > you have 3 nodes dedicated only to do cluster state/master tasks unimpeded > > > by load or anything else other than your network. Just don't run anything > > > else on them or send queries/indexing jobs to these 3 nodes. 🙂 > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e391a498-139a-4a8d-9c0c-7eb8402cfa89%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e391a498-139a-4a8d-9c0c-7eb8402cfa89%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![skik2skis](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@skik2skis](https://discuss.elastic.co/u/skik2skis)\ **Post date:** [May 30, 2014, 3:44pm UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/6 "2014-05-30T15:44:54Z")
We're still seeing node drops, and what is more bizzare is we're seeing this on a test cluster we stood up that actually has no activity on it (no reads or writes going to it). Does anyone have any additional thoughts? Here is the info from the configuration and the logs we're seeing on the drops. SC-TLS1 - 4GB Memory 1GB Heap (Master) SC-TLS2 - 4GB Memory 1GB Heap (Master) SC-TLS3 - 8GB Memory 1GB Heap (Data) SC-TLS4 - 8GB Memory 1GB Heap (Data) SC-TLS5 - 8GB Memory 1GB Heap (Data) PX-TLS3 - 8GB Memory 1GB Heap (Data) PX-TLS4 - 8GB Memory 1GB Heap (Data) PX-TLS5 - 8GB Memory 1GB Heap (Data) _Elasticsearch_ 1.0.1 _Elasticsearch Configuration Settings_ bootstrap.mlockall: true discovery.zen.ping.timeout: 15s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["10.9.84.206[9300-9400]", "10.9.84.213[9300-9400]"] action.destructive\_requires\_name: true discovery.zen.fd.ping\_interval: 30s discovery.zen.fd.ping\_timeout: 120s discovery.zen.fd.ping\_retries: 10 _Events from TLS1 (Master)_ [2014-05-26 22:09:22,953][INFO][cluster.service] [SC-TLS1] removed {[PX-TLS5][Ld8VcLgfRs2roHUWS8c6mA][PX-TLS5][inet[/10.9.64.223:9300]]{dc=PX, master=false},}, reason: zen-disco-receive(from master [[SC-TLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true}]) [2014-05-26 22:12:07,085][INFO][cluster.service] [SC-TLS1] added {[PX-TLS5][Ld8VcLgfRs2roHUWS8c6mA][PX-TLS5][inet[/10.9.64.223:9300]]{dc=PX, master=false},}, reason: zen-disco-receive(from master [[SC-TLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true}]) _Events from PX-TLS5_ [2014-05-26 22:09:37,010][INFO][discovery.zen] [PX-TLS5] master\_left [[SC-TLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true}], reason [do not exists on master, act as master failure] [2014-05-26 22:09:37,011][INFO][cluster.service] [PX-TLS5] master {new [SC-TLS1][fDW1-5P8RzWgZwGEG2BJhQ][SC-TLS1][inet[/10.9.84.206:9300]]{dc=SC, data=false, master=true}, previous [SCTLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true}}, removed {[SC-TLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true},}, reason: zen-disco-master\_failed ([SC-TLS2][8hdMizOCRz-wufVkI-IaRw][SC-tls2][inet[/10.9.84.213:9300]]{dc=SC, data=false, master=true}) [2014-05-26 22:10:07,035][INFO][discovery.zen] [PX-TLS5] master\_left [[SC-TLS1][fDW1-5P8RzWgZwGEG2BJhQ][SC-TLS1][inet[/10.9.84.206:9300]]{dc=SC, data=false, master=true}], reason [no longer master] [2014-05-26 22:10:07,036][WARN][discovery.zen] [PX-TLS5] not enough master nodes after master left (reason = no longer master), current nodes: {[PX-TLS5][Ld8VcLgfRs2roHUWS8c6mA][PX-TLS5][inet[PX-TLS5/10.9.64.223:9300]]{dc=PX, master=false},[PX-PRD-TLS3][t9ZGWrc0Qi2ASDF5te75Pw][PX-prd-tls3][inet[/10.9.64.213:9300]]{dc=PX, master=false},[SC-TLS5][NulqNMVoQiu2nu4p6w8Usg][SC-tls5][inet[/10.9.84.210:9300]]{dc=SC, master=false},[SC-TLS4][DGWDAMr9QYmN5nNjFNMyjw][SC-tls4][inet[/10.9.84.209:9300]]{dc=SC, master=false},[SC-TLS3][0QNRAMFRSgizAfWO9yxBdw][SC-tls3][inet[/10.9.84.214:9300]]{dc=SC, master=false},[PX-PRD-TLS4][4gh2\_7c2RiWY9MZQCuJtjw][PX-prd-tls4][inet[/10.9.64.214:9300]]{dc=PX, master=false},} [2014-05-26 22:10:07,037][INFO][cluster.service] [PX-TLS5] removed {[SC-TLS1][fDW1-5P8RzWgZwGEG2BJhQ][SC-TLS1][inet[/10.9.84.206:9300]]{dc=SC, data=false, master=true},[PX-PRD-TLS3][t9ZGWrc0Qi2ASDF5te75Pw][PX-prd-tls3][inet[/10.9.64.213:9300]]{dc=PX, master=false},[SC-TLS5][NulqNMVoQiu2nu4p6w8Usg][SC-tls5][inet[/10.9.84.210:9300]]{dc=SC, master=false},[SC-TLS4][DGWDAMr9QYmN5nNjFNMyjw][SC-tls4][inet[/10.9.84.209:9300]]{dc=SC, master=false},[SC-TLS3][0QNRAMFRSgizAfWO9yxBdw][SC-tls3][inet[/10.9.84.214:9300]]{dc=SC, master=false},[PX-PRD-TLS4][4gh2\_7c2RiWY9MZQCuJtjw][PX-prd-tls4][inet[/10.9.64.214:9300]]{dc=PX, master=false},}, reason: zen-disco-master\_failed ([SC-TLS1][fDW1-5P8RzWgZwGEG2BJhQ][SC-TLS1][inet[/10.9.84.206:9300]]{dc=SC, data=false, master=true}) On Monday, April 28, 2014 9:39:04 AM UTC-6, [skik...@gmail.com](mailto:skik...@gmail.com) wrote: > So far the only log message we've seen is: > zen-disco-node\_failed([CDPX-PRD-ELS4][lkquUBfHT1aXAO3-\_tCNCg][cdpx-prd-els4][inet[ > 10.9.64.142/10.9.64.142:9300]]{master=false} > [http://10.9.64.142/10.9.64.142:9300]]{master=false}](http://10.9.64.142/10.9.64.142:9300%5D%5D%7Bmaster=false%7D)), reason > failed to ping, tried [5] times, each with maximum [1m] timeout > > We have other data traversing the network that would be very sensitive to > any latency or outages, in addition to alerts that would fire off if we had > a network outage, so I am confident we don't have any network issues when > this occurs. Furthermore, we are only seeing data nodes drop, the masters > never drop. > > Is there a recommended heap size for nodes that are masters only? In > addition, any recommendations on heap size for data nodes? I assume this > could be a timeout in general during GC processes as our data nodes have > larger heaps? > > On Friday, April 25, 2014 5:49:44 PM UTC-6, Alexander Reelsen wrote: > > > Hey, > > > > is there any reason in the logfile of the master node, why it was > > deelected? (network outage as well)? Did you give your master nodes also a > > huge heap which could cause long outages during GC? > > > > --Alex > > > > On Mon, Apr 21, 2014 at 5:51 PM, [skik...@gmail.com](mailto:skik...@gmail.com) wrote: > > > > > We currently are running dedicated master nodes but I believe they are > > > also servicing queries. I can change it such that queries only hit the > > > data nodes and see if that eliminates the issue... > > > > > > On Monday, April 21, 2014 3:40:59 PM UTC-6, Binh Ly wrote: > > > > > > > Other than network, is it possible that your nodes could sometimes be > > > > overloaded such that they cannot respond immediately? If that's the case, > > > > then you can probably get 3 nodes (servers), make them master-only nodes > > > > (node.master: true, node.data: false). Set discovery.zen.minimum\_master\_nodes: > > > > 2 for those 3 nodes. And then for the rest of your other data nodes, make > > > > them non-master eligible (node.master: false, node.data: true). This way > > > > you have 3 nodes dedicated only to do cluster state/master tasks unimpeded > > > > by load or anything else other than your network. Just don't run anything > > > > else on them or send queries/indexing jobs to these 3 nodes. 🙂 > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4858a2da-5ceb-48f1-8cfe-fe460ab2dcce%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dffb154d-080d-4366-980c-b9401a9b3859%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dffb154d-080d-4366-980c-b9401a9b3859%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:25am UTC](https://discuss.elastic.co/t/max-latency-between-nodes/17115/7 "2017-07-06T01:25:43Z")