# Master node stops responding, but status is green locally **URL:** \ **Category:** Elasticsearch\ **Created:** [March 21, 2013, 1:49pm UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239 "2013-03-21T13:49:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![browe](https://avatars.discourse-cdn.com/v4/letter/b/2bfe46/32.png) [@browe](https://discuss.elastic.co/u/browe)\ **Post date:** [March 21, 2013, 1:49pm UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239/1 "2013-03-21T13:49:56Z")
I have a 4 node cluster. 1 master node with data turned off, and 3 data nodes (2 of which have the default master setting, 1 with master off). All of our application requests are directed at the master node by name - so we are calling - [node1.myelasticsearch.com/\_query](http://node1.myelasticsearch.com/_query)... We are also using ES head to monitor the server. The server runs fine for a while, but then just stops responding to requests. ES-Head no longer returns the overview page data. It does however keeping showing the cluster as green. If I log into all nodes and request heath status of the localhost node, it returns green. But the application does not respond and ES-head can not show the status on the overview page - and the structured query page doesn't load. Then I restart ES - using kill and then starting it again - we don't have it installed as a service - Everything starts responding again. There are no errors in the log. Any idea why this would be happening? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 3:54pm UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239/2 "2013-03-21T15:54:40Z")
On Thu, 2013-03-21 at 06:49 -0700, browe wrote: > I have a 4 node cluster. 1 master node with data turned off, and 3 > data nodes (2 of which have the default master setting, 1 with master > off). > All of our application requests are directed at the master node by > name - so we are calling - [node1.myelasticsearch.com/\_query](http://node1.myelasticsearch.com/_query)... > We are also using ES head to monitor the server. > The server runs fine for a while, but then just stops responding to > requests. ES-Head no longer returns the overview page data. It does > however keeping showing the cluster as green. If I log into all nodes > and request heath status of the localhost node, it returns green. But > the application does not respond and ES-head can not show the status > on the overview page - and the structured query page doesn't load. > Then I restart ES - using kill and then starting it again - we don't > have it installed as a service - Everything starts responding again. > There are no errors in the log. > Any idea why this would be happening? Are you using HTTP connections without keepalive? You may be running out of sockets. Look at netstat output for sockets in TIME\_WAIT clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![browe](https://avatars.discourse-cdn.com/v4/letter/b/2bfe46/32.png) [@browe](https://discuss.elastic.co/u/browe)\ **Post date:** [March 21, 2013, 6:58pm UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239/3 "2013-03-21T18:58:02Z")
I'm not sure I know that we are or are not use HTTP connections without keepalive. Are you asking about our application which is using RestClient to make the calls, or an ES server setting somewhere that I could check. For ES we would be using the default. Netstat does not show time\_wait on the master node right now. Should I be checking all nodes? And if they are in Timewat - how would I fix that? Thank you, Brian On Thu, Mar 21, 2013 at 11:54 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > On Thu, 2013-03-21 at 06:49 -0700, browe wrote: > > > I have a 4 node cluster. 1 master node with data turned off, and 3 > > data nodes (2 of which have the default master setting, 1 with master > > off). > > All of our application requests are directed at the master node by > > name - so we are calling - [node1.myelasticsearch.com/\_query](http://node1.myelasticsearch.com/_query)... > > We are also using ES head to monitor the server. > > The server runs fine for a while, but then just stops responding to > > requests. ES-Head no longer returns the overview page data. It does > > however keeping showing the cluster as green. If I log into all nodes > > and request heath status of the localhost node, it returns green. But > > the application does not respond and ES-head can not show the status > > on the overview page - and the structured query page doesn't load. > > Then I restart ES - using kill and then starting it again - we don't > > have it installed as a service - Everything starts responding again. > > There are no errors in the log. > > Any idea why this would be happening? > > Are you using HTTP connections without keepalive? You may be running out > of sockets. Look at netstat output for sockets in TIME\_WAIT > > clint > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/zKIv9Mk\_YkI/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/zKIv9Mk_YkI/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 8:09pm UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239/4 "2013-03-21T20:09:54Z")
Sorry, to be clearer: a typical issue that stops requests getting through to Elasticsearch, is that the HTTP client you are using does not use keepalive, which means that it opens a new socket for every connection. The kernel keeps old sockets around for a while, before finally clearing them out. There is a limit as to how many sockets can be opened, after which you have to wait for old connections to be removed before you can open new sockets. You are using your client to talk to the master node, so I would look for TIME\_WAITs on the master node and on the clients. As for fixing it, you can google for settings that will change how many open sockets your kernel will allow, or switch to using an HTTP client which supports keepalive clint On Thu, Mar 21, 2013 at 7:58 PM, Brian Rowe [browe@perceivant.com](mailto:browe@perceivant.com) wrote: > I'm not sure I know that we are or are not use HTTP connections without > keepalive. Are you asking about our application which is using RestClient > to make the calls, or an ES server setting somewhere that I could check. > For ES we would be using the default. > Netstat does not show time\_wait on the master node right now. Should I be > checking all nodes? > And if they are in Timewat - how would I fix that? > > Thank you, > > Brian > > On Thu, Mar 21, 2013 at 11:54 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > > > On Thu, 2013-03-21 at 06:49 -0700, browe wrote: > > > > > I have a 4 node cluster. 1 master node with data turned off, and 3 > > > data nodes (2 of which have the default master setting, 1 with master > > > off). > > > All of our application requests are directed at the master node by > > > name - so we are calling - [node1.myelasticsearch.com/\_query](http://node1.myelasticsearch.com/_query)... > > > We are also using ES head to monitor the server. > > > The server runs fine for a while, but then just stops responding to > > > requests. ES-Head no longer returns the overview page data. It does > > > however keeping showing the cluster as green. If I log into all nodes > > > and request heath status of the localhost node, it returns green. But > > > the application does not respond and ES-head can not show the status > > > on the overview page - and the structured query page doesn't load. > > > Then I restart ES - using kill and then starting it again - we don't > > > have it installed as a service - Everything starts responding again. > > > There are no errors in the log. > > > Any idea why this would be happening? > > > > Are you using HTTP connections without keepalive? You may be running out > > of sockets. Look at netstat output for sockets in TIME\_WAIT > > > > clint > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/zKIv9Mk\_YkI/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/zKIv9Mk_YkI/unsubscribe?hl=en-US) > > . > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/master-node-stops-responding-but-status-is-green-locally/11239/5 "2017-07-06T02:45:13Z")