I have setup a elasticsearch cluster with 3 master node and 2 data node on k8s. While start master node , it failed.
System config:
ulimit -n 65535
ulimit -l unlimited
sysctl -w vm.max_map_count=262144
sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=60
Running command with jvm options: -Xmx2g -Xms2g
/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-9298489965720023744 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xmx2g -Xms2g -XX:MaxDirectMemorySize=1073741824 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Enode.name=ems-search-000-master-1 -Enode.transform=false -Ecluster.initial_master_nodes=ems-search-000-master-0,ems-search-000-master-1,ems-search-000-master-2, -Enetwork.host=0.0.0.0 -Enode.remote_cluster_client=false -Enode.ingest=false -Enode.data=false -Ediscovery.seed_hosts=ems-search-000-master-headless -Ecluster.name=ems-search-000 -Enode.ml=false -Enode.master=true
We first deployed a master on aws m5.large instance with 8GB RAM, it failed.
While we deployed it on aws m5.xlarge instance with 16GB RAM, it worked.
Got error:
vm.max_map_count = 262144
vm.overcommit_memory = 2
vm.overcommit_ratio = 60
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000aaaa0000, 1431699456, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1431699456 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/elasticsearch/hs_err_pid19.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000aaaa0000, 1431699456, 0) failed; error='Not enough space' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000aaaa0000, 1431699456, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1431699456 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/elasticsearch/hs_err_pid131.log
Why master node can't start on instance with 8GB RAM, I think the memory is enough.