ElasticSearch 1.7.3 crashed and doesn't restart - no logs

https://gist.github.com/ThomasdOtreppe/5a3febe7405872759f4e

Seems like the file was truncated. Probably need to use google drive or something similar.

http://demo.ovh.eu/en/3f3613900a3ca3a7876db23a52f6070e/

Is it any useful? Anything else I can provide to help diagnose the issue?

Any update?

If anybody at Elastic is interested, I can give access to the server through a TeamViewer session (or something similar).

Hi,

I'm not sure if this is because we're trying to run Elasticsearch in strace or if this is the issue but this error occurs:

[pid 25261] write(2, "Java HotSpot(TM) 64-Bit Server VM warning: ", 43) = 43
[pid 25261] write(2, "INFO: os::commit_memory(0x000000031f330000, 19877658624, 0) failed; error='Cannot allocate memory' "..., 109) = 109
[pid 25261] write(2, "\n", 1)           = 1
[pid 25261] write(1, "#\n", 2)          = 2
[pid 25261] write(1, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77
[pid 25261] write(1, "# Native memory allocation ", 27) = 27
[pid 25261] write(1, "(mmap) failed to map ", 21) = 21
[pid 25261] write(0, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77

I suggest you check what you've set ES_HEAP_SIZE to or whatever you're setting as -Xmx and -Xms. From the strace it shows:

[pid 25260] execve("/usr/lib/jvm/java-8-oracle//bin/java", ["/usr/lib/jvm/java-8-oracle//bin/java", "-Xms19g", "-Xmx19g",

I'd confirm you have sufficient amount of memory for this heap size or try reducing your heap size.

1 Like

Thanks to the help on IRC and here, decreasing the amount of memory allocated to ES worked (changing ES_HEAP).

I haven't changed anything in Xmx and Xms.

Problem seems possibly related to growth of memory use by Logstash on the same server.

"input: a few rabbitmq, one tcp, 2 files. Filter: geoip. Output: ES"

Install LS 2.1.1 when it drops.

Is there any way to monitor the memory usage of the process and the system and if it reaches a certain point, warn the user in the logs that it will soon fail?

When a JVM mysteriously dies, you can find an error log in /tmp/hs_err_pid*.log by default, or you can set this with -XX:ErrorFile=path. The reason that no logging is showing is because of how the service is configured in systemd. The key is the StandardOutput configuration:

# Connects standard output to /dev/null
StandardOutput=null

Change null to journal and reload the dameon. Now when the JVM dies it will show the errors in the system journal:

# journalctl

gives

Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006fad30000, 4207738880, 0) failed; error='Cannot allocate memory' (errno=12)
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: #
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # There is insufficient memory for the Java Runtime Environment to continue.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # Native memory allocation (malloc) failed to allocate 4207738880 bytes for committing reserved memory.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # An error report file with more information is saved as:
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # /tmp/hs_err_pid16129.log

Please check the man pages for systemd.exec, systemd.journald, and journalctl for additional configuration options.