OOM Error

Elastic Stack Elasticsearch
#1

Hi Team,

I am using ElasticSearch 7.6.2, sometimes when we request query I am getting OOM (Out Of Memory) may I know how to tackle this issue?

Hardware environment as follows:
Server capacity
vCPUs: 8
RAM: 32GB
Storage: 100GB

ES Health
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 12,
"active_shards" : 12,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 12,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0

Elastic Version Details
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "***************************",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},

memory allocated to ES is 20GB, but still getting OOM error sometimes, need to Restart ES every time. :frowning:

#2

Assuming Elasticsearch is the only thing running on the host it is recommended you give no more than 50% of available memory to the heap. You seem to have relatively little data in the cluster so it is surprising you are experiencing OOM.

What kind of workload is the cluster under?

Do you have any non-default cluster or index settings in place?

Do you have any third-party plugins installed?

Is there anything else running on that host?

#3

Hi @Christian_Dahlqvist,

Thank you for your response.

Here are some details:

We tried to allocate only 50% of server capacity, but OOM error frequency is height, if I allocate more memory then it will reduce displaying OOM frequency

What kind of workload is the cluster under?
It is very Minimal workload, you can consider 2 to 3L data/day.

Do you have any non-default cluster or index settings in place?
No, I don't have any non-default cluster or index setting.
Index setting as follows:
{"settings":
{"index":{"creation_date":"1560427000684","
number_of_shards":"1",
"number_of_replicas":"1",
"uuid":"*************",
"version":{"created":"7010099","upgraded":"7060299"},
"provided_name":"********"}}}

Do you have any third-party plugins installed?
No, we don't have any third-party plugins installed.

Is there anything else running on that host?
yes, we have Web application hosted in the same server, but that Utilization is minimal.

#4

Can you show the full output of the cluster stats API?

What is the OOM message? Do you see any long or slow GC in the logs?

#5

@Christian_Dahlqvist

I checked with GC Log, but I haven't seen any long or Slow logs

Here is the sample OOM Error Log from ES

Elasticsearch Log
"type": "server",
"timestamp": "2020-08-31T16:28:42,673Z",
"level": "INFO",
"component": "o.e.n.Node",
"cluster.name": "elasticsearch",
"node.name": "ip----",
"message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=100, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-1422291560869168296, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=10737418240, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]"

#6

Have you changed or updated any JVM options?

#7

@Christian_Dahlqvist

Recently we modified only Xms and Xmx in JVM.options

JVM.options as follows
##JVM configuration##
#Xmx represents the maximum size of total heap space
-Xms20g
-Xmx20g
#GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=100
8-13:-XX:+UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
#JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}
##heap dumps
#generate a heap dump when an allocation from the Java heap fails
#heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
#specify an alternative path for heap dumps; ensure the directory exists and
#has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch
#specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
##JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
#JDK 9+ GC logging
9-:Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

#8

@Christian_Dahlqvist

Any Update..?