# ElasticSearch 1.7.3 crashed and doesn't restart - no logs

**URL:** <https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080>\
**Category:** Elasticsearch\
**Created:** [December 1, 2015, 5:21pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080 "2015-12-01T17:21:02Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 3, 2015, 4:20pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/21 "2015-12-03T16:20:14Z")

</div>

[https://gist.github.com/ThomasdOtreppe/5a3febe7405872759f4e](https://gist.github.com/ThomasdOtreppe/5a3febe7405872759f4e)

---

<div class="post-metadata">

**Author:** ![msimos](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@msimos](https://discuss.elastic.co/u/msimos)\
**Post date:** [December 3, 2015, 4:55pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/22 "2015-12-03T16:55:16Z")

</div>

Seems like the file was truncated. Probably need to use google drive or something similar.

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 3, 2015, 5:11pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/23 "2015-12-03T17:11:46Z")

</div>

[http://demo.ovh.eu/en/3f3613900a3ca3a7876db23a52f6070e/](http://demo.ovh.eu/en/3f3613900a3ca3a7876db23a52f6070e/)

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 4, 2015, 9:28pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/24 "2015-12-04T21:28:36Z")

</div>

Is it any useful? Anything else I can provide to help diagnose the issue?

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 7, 2015, 5:01pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/25 "2015-12-07T17:01:35Z")

</div>

Any update?

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 7, 2015, 11:46pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/26 "2015-12-07T23:46:32Z")

</div>

If anybody at Elastic is interested, I can give access to the server through a TeamViewer session (or something similar).

---

<div class="post-metadata">

**Author:** ![msimos](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@msimos](https://discuss.elastic.co/u/msimos)\
**Post date:** [December 8, 2015, 12:27am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/27 "2015-12-08T00:27:11Z")

</div>

Hi,

I'm not sure if this is because we're trying to run Elasticsearch in strace or if this is the issue but this error occurs:

```auto
[pid 25261] write(2, "Java HotSpot(TM) 64-Bit Server VM warning: ", 43) = 43
[pid 25261] write(2, "INFO: os::commit_memory(0x000000031f330000, 19877658624, 0) failed; error='Cannot allocate memory' "..., 109) = 109
[pid 25261] write(2, "\n", 1) = 1
[pid 25261] write(1, "#\n", 2) = 2
[pid 25261] write(1, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77
[pid 25261] write(1, "# Native memory allocation ", 27) = 27
[pid 25261] write(1, "(mmap) failed to map ", 21) = 21
[pid 25261] write(0, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77

```

I suggest you check what you've set ES\_HEAP\_SIZE to or whatever you're setting as -Xmx and -Xms. From the strace it shows:

```auto
[pid 25260] execve("/usr/lib/jvm/java-8-oracle//bin/java", ["/usr/lib/jvm/java-8-oracle//bin/java", "-Xms19g", "-Xmx19g",

```

I'd confirm you have sufficient amount of memory for this heap size or try reducing your heap size.

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 8, 2015, 12:36am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/28 "2015-12-08T00:36:08Z")

</div>

Thanks to the help on IRC and here, decreasing the amount of memory allocated to ES worked (changing ES\_HEAP).

I haven't changed anything in Xmx and Xms.

---

<div class="post-metadata">

**Author:** ![GlenRSmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenrsmith/32/44925_2.png) [@GlenRSmith](https://discuss.elastic.co/u/GlenRSmith)\
**Post date:** [December 8, 2015, 2:03am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/29 "2015-12-08T02:03:29Z")

</div>

Problem seems possibly related to growth of memory use by Logstash on the same server.

"input: a few rabbitmq, one tcp, 2 files. Filter: geoip. Output: ES"

Install LS 2.1.1 when it drops.

---

<div class="post-metadata">

**Author:** ![thomas.dotreppe](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\
**Post date:** [December 8, 2015, 3:49pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/30 "2015-12-08T15:49:03Z")

</div>

Is there any way to monitor the memory usage of the process and the system and if it reaches a certain point, warn the user in the logs that it will soon fail?

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [December 8, 2015, 6:27pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/31 "2015-12-08T18:27:38Z")

</div>

When a JVM mysteriously dies, you can find an error log in `/tmp/hs_err_pid*.log` by default, or you can set this with `-XX:ErrorFile=path`. The reason that no logging is showing is because of how the service is configured in `systemd`. The key is the [`StandardOutput` configuration](https://github.com/elastic/elasticsearch/blob/6f996e9e26ae05440aa8d374d5cedcc3752ebfec/distribution/src/main/packaging/systemd/elasticsearch.service#L29-L30):

```auto
# Connects standard output to /dev/null
StandardOutput=null

```

Change `null` to `journal` and reload the dameon. Now when the JVM dies it will show the errors in the system journal:

```auto
# journalctl

```

gives

```auto
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006fad30000, 4207738880, 0) failed; error='Cannot allocate memory' (errno=12)
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: #
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # There is insufficient memory for the Java Runtime Environment to continue.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # Native memory allocation (malloc) failed to allocate 4207738880 bytes for committing reserved memory.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # An error report file with more information is saved as:
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # /tmp/hs_err_pid16129.log

```

Please check the man pages for `systemd.exec`, `systemd.journald`, and `journalctl` for additional configuration options.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:32pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080/32 "2017-07-05T23:32:32Z")

</div>

Closed on July 5, 2017, 11:32pm UTC



[Previous page](https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080.md?page=1)
