# Elasticsearch terminated unexpectedly

**URL:** <https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894>\
**Category:** Elasticsearch\
**Created:** [January 3, 2018, 9:08am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894 "2018-01-03T09:08:52Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 3, 2018, 9:08am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/1 "2018-01-03T09:08:52Z")

</div>

I restarted elasticsearch service several times and each time it initialized the kibana index for a while and then terminated unexpectedly. Is there anything I can do to fix this problem?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 3, 2018, 9:21am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/2 "2018-01-03T09:21:04Z")

</div>

Please show your logs, as well as your OS, JVM and Elasticsearch version.

---

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 3, 2018, 9:50am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/3 "2018-01-03T09:50:43Z")

</div>

I am using windows server 2012 R2, JVM 8 and Elasticsearch 6. What is the path of the logs to be collected?

---

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 4, 2018, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/4 "2018-01-04T07:49:11Z")

</div>

The log is a bit large. Can I upload other than image file?  
I found that some java\_pidxxxx.hprof files are being generated and they quickly fill up the disk space. Anything wrong with that?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 4, 2018, 9:06am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/5 "2018-01-04T09:06:59Z")

</div>

> [@Terry](#):
>
> I found that some java\_pidxxxx.hprof files are being generated and they quickly fill up the disk space. Anything wrong with that?

They are crashes that are logged to disk.

Just show the last few parts of the log, or use gist/pastebin/etc.

---

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 4, 2018, 9:48am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/6 "2018-01-04T09:48:03Z")

</div>

[2018-01-04T03:07:54,846][WARN][o.e.g.MetaStateService] [OANlHwb] [[winlogbeat-6.0.0-2017.08.04/81UkYv3TSG2GxAAewfY5tQ]]: failed to write index state  
java.io.IOException: There is not enough space on the disk  
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]  
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:75) ~[?:?]  
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]  
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]  
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) ~[?:?]  
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.writeFully(Channels.java:101) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.access$000(Channels.java:61) ~[?:1.8.0\_151]  
at java.nio.channels.Channels$1.write(Channels.java:174) ~[?:1.8.0\_151]  
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0\_151]  
at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]  
at org.apache.lucene.codecs.CodecUtil.writeCRC(CodecUtil.java:548) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]  
at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:393) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]  
at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:140) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:125) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.GatewayMetaState.applyClusterState(GatewayMetaState.java:180) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.Gateway.applyClusterState(Gateway.java:181) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:495) ~[elasticsearch-6.0.0.jar:6.0.0]  
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0\_151]  
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:492) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:479) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:158) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.0.0.jar:6.0.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151]  
Suppressed: java.io.IOException: There is not enough space on the disk  
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]  
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:75) ~[?:?]  
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]  
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]  
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) ~[?:?]  
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.writeFully(Channels.java:101) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.access$000(Channels.java:61) ~[?:1.8.0\_151]  
at java.nio.channels.Channels$1.write(Channels.java:174) ~[?:1.8.0\_151]  
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0\_151]  
at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:68) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]  
at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:141) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:125) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.GatewayMetaState.applyClusterState(GatewayMetaState.java:180) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.Gateway.applyClusterState(Gateway.java:181) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:495) ~[elasticsearch-6.0.0.jar:6.0.0]  
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0\_151]  
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:492) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:479) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:158) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.Pri

---

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 4, 2018, 9:48am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/7 "2018-01-04T09:48:44Z")

</div>

ioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.0.0.jar:6.0.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151]  
Suppressed: java.io.IOException: There is not enough space on the disk  
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]  
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:75) ~[?:?]  
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]  
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]  
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) ~[?:?]  
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.writeFully(Channels.java:101) ~[?:1.8.0\_151]  
at java.nio.channels.Channels.access$000(Channels.java:61) ~[?:1.8.0\_151]  
at java.nio.channels.Channels$1.write(Channels.java:174) ~[?:1.8.0\_151]  
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0\_151]  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0\_151]  
at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0\_151]  
at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:70) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]  
at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:141) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:125) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.GatewayMetaState.applyClusterState(GatewayMetaState.java:180) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.gateway.Gateway.applyClusterState(Gateway.java:181) ~[elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:495) ~[elasticsearch-6.0.0.jar:6.0.0]  
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0\_151]  
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:492) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:479) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:158) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.0.0.jar:6.0.0]  
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.0.0.jar:6.0.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_151]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_151]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_151]  
[2018-01-04T03:08:06,317][WARN][o.e.c.a.s.ShardStateAction] [OANlHwb] [winlogbeat-6.0.0-2017.07.12][1] received shard failed for shard id [[winlogbeat-6.0.0-2017.07.12][1]], allocation id [IMs2AC\_7RLe6TIHA0vIfrw], primary term [0], message [master {OANlHwb}{OANlHwb4Q0WNQb6XQx1x0w}{LitHRKi2TveqK6m1k-\_N4w}{10.99.210.12}{10.99.210.12:9300} has not removed previously failed shard. resending shard failure]

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 4, 2018, 9:45pm UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/8 "2018-01-04T21:45:00Z")

</div>

> [@Terry](#):
>
> There is not enough space on the disk

You may want to check this.

---

<div class="post-metadata">

**Author:** ![Terry](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Terry](https://discuss.elastic.co/u/Terry)\
**Post date:** [January 10, 2018, 8:36am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/9 "2018-01-10T08:36:27Z")

</div>

If I want to move the directory for data and logs, how can I configure it?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 10, 2018, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/10 "2018-01-10T08:55:43Z")

</div>

[https://www.elastic.co/guide/en/elasticsearch/reference/6.1/modules-node.html#\_node\_data\_path\_settings](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/modules-node.html#_node_data_path_settings) should help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2018, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-terminated-unexpectedly/113894/11 "2018-02-07T08:55:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.