Cannot allocate memory error despite enough memory

[thefourtheye]

My machine has a 24G RAM.

$ free -h
             total       used       free     shared    buffers     cached
Mem:           23G       257M        23G         0B        12M        39M
-/+ buffers/cache:       206M        23G
Swap:         2.0G        21M       2.0G

And

$ echo $ES_HEAP_SIZE
12g

But still when I start elasticsearch it says that

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004e9990000, 12187009024, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12187009024 bytes for committing reserved memory.

I use elasticsearch 2.4.4. Following are the system configurations which might help in find the problem.

$ tail /etc/sysctl.conf

#Controls the maximum number of map count
vm.max_map_count=262144
vm.min_free_kbytes=24794496
vm.swappiness=1
fs.file-max = 100000

and

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 96030
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 210000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[thefourtheye]

bump

[spinscale (Alexander Reelsen)]

please be aware that this is a public forum without any SLAs when bumping threads.

Have you enabled memory locking in Elasticsearch? I see it is set to unlimited in the ulimit output, but there is more than one configuration option to ensure this (depends on the distribution). If it is enabled, disable it, to check if that is the cuplrit.

Any other changes you did? JVM options, startup options or general configuration?

--Alex

[thefourtheye]

please be aware that this is a public forum without any SLAs when bumping threads.

Oh, sorry about that.

I am writing to say that this problem is solved. This elasticsearch instance is running in a VM, and it had a specific configuration which made the operating system to keep all the 24GB free it seems.

/proc/sys/vm/min_free_kbytes file had the value 24794496. I changed it to a much lower value, and it started working.

[spinscale (Alexander Reelsen)]

out of curiosity and to help users in the future. Who was configuring this setting to that value? The operating system with some automation or the sysadmin?

[thefourtheye]

I am not very sure. This was a brand new VM, created 15 mins before I tried to install elasticsearch. So I am guessing, when it was created some configuration value was wrong, but I could be wrong. Sorry, thats the best information I could give.