# My cluster stopped automatically, what's the problem? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2017, 2:13am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807 "2017-04-19T02:13:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [April 19, 2017, 2:13am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/1 "2017-04-19T02:13:13Z")
Hi all, Is there any problem causes the cluster shutdown automatically? All the ten nodes shutdown this morning. We found this problem in 16th, and today it appears again. The logs i can found around shutdown including: The first node: [There is no log befor this one today] [2017-04-19T05:39:45,428][INFO][o.e.n.Node] [index188] stopping ... [2017-04-19T05:39:49,789][DEBUG][o.e.a.a.i.f.TransportForceMergeAction] [index18 8] [indices:admin/forcemerge] failed to execute operation for shard [[tj\_news\_re cent][2], node[5at14jo6Qn-GVe4nGw\_jsw], [R], s[STARTED], a[id=38eb4B-iRmWARSZmOQ inkA]] java.io.IOException: background merge hit exception: \_5pej8(6.2.0):C2833847/2679 34:delGen=13 \_5qwoc(6.2.0):c64302/2:delGen=1 \_5q4vr(6.2.0):c22749/115:delGen=4 i nto \_5qwp4 [maxNumSegments=1] [ABORTED] This log sames caused by the stopping request. After that, the node stopped. The second: Initially, some events about nodes left. (Since the first node is stopped) [2017-04-19T06:10:11,774][INFO][o.e.n.Node] [index187] stopping ... [2017-04-19T06:10:11,929][DEBUG][o.e.a.a.i.f.TransportForceMergeAction] [index18 7] failed to execute [indices:admin/forcemerge] on node [4jXUkkeERDCFf09LADlHBg] org.elasticsearch.transport.TransportException: transport stopped, action: indic es:admin/forcemerge[n]... [2017-04-19T06:10:11,929][ERROR][n.u.c.D.rejectedExecution] Failed to submit a l istener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated followed with some similar logs, and finally stopped. The third: [2017-04-19T01:20:19,079][INFO][o.e.m.j.JvmGcMonitorService] [index185] [gc][19 764] overhead, spent [473ms] collecting in the last [1s] [2017-04-19T02:47:13,759][INFO][o.e.m.j.JvmGcMonitorService] [index185] [gc][24 975] overhead, spent [251ms] collecting in the last [1s] ...Some node disconnected message [2017-04-19T06:10:08,317][INFO][o.e.n.Node] [index185] stopping ... [2017-04-19T06:10:10,472][ERROR][n.u.c.D.rejectedExecution] Failed to submit a l istener notification task. Event loop shut down? Similar logs with the previous one, and stopped. Anyway i didn't find any useful information from the log. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 19, 2017, 2:25am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/2 "2017-04-19T02:25:35Z")
What version? ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [April 19, 2017, 2:40am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/3 "2017-04-19T02:40:18Z")
5.0.0. BTW: is it worth to update from 5.0.0 to the current version? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 19, 2017, 2:41am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/4 "2017-04-19T02:41:40Z")
It's always worth updating 🙂 However in this case the only way for this to happen is for something to stop the service, there are no APIs in ES 5.X that can shutdown the nodes. You should look in your OS logs to see if there is anything that correlates. ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [April 19, 2017, 3:13am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/5 "2017-04-19T03:13:47Z")
Thank you. That's also what i assumed. And i added a audit to log all the kill commands this morning. Hope it can help me... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 17, 2017, 3:26am UTC](https://discuss.elastic.co/t/my-cluster-stopped-automatically-whats-the-problem/82807/6 "2017-05-17T03:26:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.