# Node will not shut down **URL:** \ **Category:** Elasticsearch\ **Created:** [January 21, 2014, 9:22pm UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349 "2014-01-21T21:22:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Brad\_Jordan\_2](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Brad\_Jordan\_2](https://discuss.elastic.co/u/Brad_Jordan_2)\ **Post date:** [January 21, 2014, 9:22pm UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/1 "2014-01-21T21:22:50Z")
I have a 4 node ES cluster. I am adding new docs at a rate of about 3k/sec. After an hour my cluster status turns to yellow. I see that node 2 is no longer part of the cluster when I curl the "\_cluster/state" endpoint. Hitting the "\_cluster/health" endpoint reveals that I have 872 unassigned shards. I login to node 2 and find ES running. I want to restart ES on node 2 to see if it will rejoin my cluster. I issue the command: "curl -XPOST '[http://localhost:9200/\_cluster/nodes/\_local/\_shutdown](http://localhost:9200/_cluster/nodes/_local/_shutdown)' " and get back curl: (56) Failure when receiving data from the peer. I try this several more times and ES will still not shut down on node 2. I believe at this point if I kill -9 the process the cluster will move from "yellow" to "red". How do I gracefully recover from this situation? Is it possible? Thanks, Brad -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3e8c18e8-6f7c-48b7-a7a4-ed0c2a072871%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3e8c18e8-6f7c-48b7-a7a4-ed0c2a072871%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ben\_Hundley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben_hundley/32/1853_2.png) [@Ben\_Hundley](https://discuss.elastic.co/u/Ben_Hundley)\ **Post date:** [January 21, 2014, 9:54pm UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/2 "2014-01-21T21:54:52Z")
2 questions: 1. What size servers are you using? Knowing how much RAM and # cores would be very helpful. 2. Definitely sounds like a massive load. Are you going to continually be inserting 3k docs per sec? ~260mil documents a day? ---
**Author:** ![Brad\_Jordan\_2](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Brad\_Jordan\_2](https://discuss.elastic.co/u/Brad_Jordan_2)\ **Post date:** [January 21, 2014, 10:42pm UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/3 "2014-01-21T22:42:51Z")
This is a DEV env. I've got 24G of RAM on all 4 machines. 12G for ES and 12G for the OS. I believe the machines are quad core HP Z-800's. I will not be inserting at this rate very often. My question is more operational. How do you recover from the place I am in? If I kill -9 the ES process on node 2 I believe I will put my cluster in the red state. I did get into this unhappy spot once before. After trying to shut down ES on node 2 I eventually kill -9'd it. At that point my cluster was in the red state and unable to service requests. The "unassigned\_shards" number was not changing. I have daily indexes so I simply deleted the most recent daily index and rebuilt it. At this point my cluster had all 4 nodes and was green again. In production this approach is not popular with mgmt. so I'm trying to understand a less heavy handed approach 😉 -Brad On Tuesday, January 21, 2014 2:54:52 PM UTC-7, Ben Hundley wrote: > 2 questions: > > 1. What size servers are you using? Knowing how much RAM and # cores > would > be very helpful. > > 2. Definitely sounds like a massive load. Are you going to continually be > inserting 3k docs per sec? ~260mil documents a day? > > * * * > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html](http://elasticsearch-users.115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3c667258-6c16-4557-8816-bd98ef0069b5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c667258-6c16-4557-8816-bd98ef0069b5%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Brad\_Jordan\_2](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Brad\_Jordan\_2](https://discuss.elastic.co/u/Brad_Jordan_2)\ **Post date:** [January 21, 2014, 10:55pm UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/4 "2014-01-21T22:55:52Z")
Just an update... I waited for the "unassigned\_shards" number to reach zero at which point the cluster\_state reported GREEN but still only had 3 nodes. I was then able to execute the curl to shut down the node 2 and then restart it. It joined the cluster again and everyone was happy. I guess the moral of the story is to just wait and ES will fix itself? Patience is a virtue? Not sure but ES did eventually fix itself 🙂 -Brad On Tuesday, January 21, 2014 3:42:51 PM UTC-7, Brad Jordan wrote: > This is a DEV env. I've got 24G of RAM on all 4 machines. 12G for ES and > 12G for the OS. I believe the machines are quad core HP Z-800's. > > I will not be inserting at this rate very often. My question is more > operational. How do you recover from the place I am in? If I kill -9 the ES > process on node 2 I believe I will put my cluster in the red state. > > I did get into this unhappy spot once before. After trying to shut down ES > on node 2 I eventually kill -9'd it. At that point my cluster was in the > red state and unable to service requests. The "unassigned\_shards" number > was not changing. I have daily indexes so I simply deleted the most recent > daily index and rebuilt it. At this point my cluster had all 4 nodes and > was green again. In production this approach is not popular with mgmt. so > I'm trying to understand a less heavy handed approach 😉 > > -Brad > > On Tuesday, January 21, 2014 2:54:52 PM UTC-7, Ben Hundley wrote: > > > 2 questions: > > > > 1. What size servers are you using? Knowing how much RAM and # cores > > would > > be very helpful. > > > > 2. Definitely sounds like a massive load. Are you going to continually > > be > > inserting 3k docs per sec? ~260mil documents a day? > > > > * * * > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html](http://elasticsearch-users.115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a13cec60-5329-46f1-8228-03ebad5f5cee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a13cec60-5329-46f1-8228-03ebad5f5cee%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [January 22, 2014, 12:24am UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/5 "2014-01-22T00:24:08Z")
Brad, What version of ES? What was ES doing on the problem node? (Hot threads call/ log file/ strace). Any related OS info( was it io bound? ) If it was really hung, I am not sure why the shutdown would work after moving the shards off it ( I.e. cluster was green...) ...it sounds to me like it was too busy doing something... The other nodes had already decided your problem node was out of the cluster , right? On 22/01/2014 9:55 AM, "Brad Jordan" [climberbrad@gmail.com](mailto:climberbrad@gmail.com) wrote: > Just an update... I waited for the "unassigned\_shards" number to reach > zero at which point the cluster\_state reported GREEN but still only had 3 > nodes. I was then able to execute the curl to shut down the node 2 and then > restart it. It joined the cluster again and everyone was happy. I guess the > moral of the story is to just wait and ES will fix itself? Patience is a > virtue? Not sure but ES did eventually fix itself 🙂 > > -Brad > > On Tuesday, January 21, 2014 3:42:51 PM UTC-7, Brad Jordan wrote: > > > This is a DEV env. I've got 24G of RAM on all 4 machines. 12G for ES and > > 12G for the OS. I believe the machines are quad core HP Z-800's. > > > > I will not be inserting at this rate very often. My question is more > > operational. How do you recover from the place I am in? If I kill -9 the ES > > process on node 2 I believe I will put my cluster in the red state. > > > > I did get into this unhappy spot once before. After trying to shut down > > ES on node 2 I eventually kill -9'd it. At that point my cluster was in the > > red state and unable to service requests. The "unassigned\_shards" number > > was not changing. I have daily indexes so I simply deleted the most recent > > daily index and rebuilt it. At this point my cluster had all 4 nodes and > > was green again. In production this approach is not popular with mgmt. so > > I'm trying to understand a less heavy handed approach 😉 > > > > -Brad > > > > On Tuesday, January 21, 2014 2:54:52 PM UTC-7, Ben Hundley wrote: > > > > > 2 questions: > > > > > > 1. What size servers are you using? Knowing how much RAM and # cores > > > would > > > be very helpful. > > > > > > 2. Definitely sounds like a massive load. Are you going to continually > > > be > > > inserting 3k docs per sec? ~260mil documents a day? > > > > > > * * * > > > > > > -- > > > View this message in context: [http://elasticsearch-users](http://elasticsearch-users). > > > [115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html](http://115913.n3.nabble.com/Node-will-not-shut-down-tp4047940p4047942.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a13cec60-5329-46f1-8228-03ebad5f5cee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a13cec60-5329-46f1-8228-03ebad5f5cee%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACj2-4JgZpGfAy%2BgFYMTsAGG8sjMPSWn1%3D4059QrjqHU6KvXPg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACj2-4JgZpGfAy%2BgFYMTsAGG8sjMPSWn1%3D4059QrjqHU6KvXPg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:55am UTC](https://discuss.elastic.co/t/node-will-not-shut-down/15349/6 "2017-07-06T01:55:29Z")