Seems like the file was truncated. Probably need to use google drive or something similar.
Is it any useful? Anything else I can provide to help diagnose the issue?
Any update?
If anybody at Elastic is interested, I can give access to the server through a TeamViewer session (or something similar).
Hi,
I'm not sure if this is because we're trying to run Elasticsearch in strace or if this is the issue but this error occurs:
[pid 25261] write(2, "Java HotSpot(TM) 64-Bit Server VM warning: ", 43) = 43
[pid 25261] write(2, "INFO: os::commit_memory(0x000000031f330000, 19877658624, 0) failed; error='Cannot allocate memory' "..., 109) = 109
[pid 25261] write(2, "\n", 1) = 1
[pid 25261] write(1, "#\n", 2) = 2
[pid 25261] write(1, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77
[pid 25261] write(1, "# Native memory allocation ", 27) = 27
[pid 25261] write(1, "(mmap) failed to map ", 21) = 21
[pid 25261] write(0, "# There is insufficient memory for the Java Runtime Environment to continue.\n", 77) = 77
I suggest you check what you've set ES_HEAP_SIZE to or whatever you're setting as -Xmx and -Xms. From the strace it shows:
[pid 25260] execve("/usr/lib/jvm/java-8-oracle//bin/java", ["/usr/lib/jvm/java-8-oracle//bin/java", "-Xms19g", "-Xmx19g",
I'd confirm you have sufficient amount of memory for this heap size or try reducing your heap size.
Thanks to the help on IRC and here, decreasing the amount of memory allocated to ES worked (changing ES_HEAP).
I haven't changed anything in Xmx and Xms.
Problem seems possibly related to growth of memory use by Logstash on the same server.
"input: a few rabbitmq, one tcp, 2 files. Filter: geoip. Output: ES"
Install LS 2.1.1 when it drops.
Is there any way to monitor the memory usage of the process and the system and if it reaches a certain point, warn the user in the logs that it will soon fail?
When a JVM mysteriously dies, you can find an error log in /tmp/hs_err_pid*.log by default, or you can set this with -XX:ErrorFile=path. The reason that no logging is showing is because of how the service is configured in systemd. The key is the StandardOutput configuration:
# Connects standard output to /dev/null
StandardOutput=null
Change null to journal and reload the dameon. Now when the JVM dies it will show the errors in the system journal:
# journalctl
gives
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006fad30000, 4207738880, 0) failed; error='Cannot allocate memory' (errno=12)
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: #
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # There is insufficient memory for the Java Runtime Environment to continue.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # Native memory allocation (malloc) failed to allocate 4207738880 bytes for committing reserved memory.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # An error report file with more information is saved as:
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # /tmp/hs_err_pid16129.log
Please check the man pages for systemd.exec, systemd.journald, and journalctl for additional configuration options.