# Elastic-Search Heap issues **URL:** \ **Category:** Elasticsearch\ **Created:** [December 6, 2018, 7:16pm UTC](https://discuss.elastic.co/t/elastic-search-heap-issues/159790 "2018-12-06T19:16:49Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![suresh\_itha](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@suresh\_itha](https://discuss.elastic.co/u/suresh_itha)\ **Post date:** [December 6, 2018, 7:16pm UTC](https://discuss.elastic.co/t/elastic-search-heap-issues/159790/1 "2018-12-06T19:16:50Z")
Hi Team, We are facing heap issues in elastic search 1.7.3 version for all the data nodes. Please find the below steps what we have implemented in my cluster. Step-1: We have total 16 data nodes and each node is having 3 instances (data1, data2 and data3) total we have 48 instances and 3 masters+ 16 separate ingest(search) nodes. All the data nodes are bare metals and each node is having 7.1TB disk. Filesystem Size Used Avail Use% Mounted on /dev/sdi2 132G 16G 110G 13% / devtmpfs 252G 0 252G 0% /dev tmpfs 252G 0 252G 0% /dev/shm tmpfs 252G 26M 252G 1% /run tmpfs 252G 0 252G 0% /sys/fs/cgroup /dev/mapper/Source--ES--eph-volume--367978823--14 7.0T 642G 6.4T 9% /app Step-2: Please find the ES process configuration elastic+ 13580 1 48 Dec05 ? 11:42:26 /bin/java -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseG1GC -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+UseCompressedOops -XX:MaxGCPauseMillis=200 -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=20 -XX:+UseStringDeduplication -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3335 -Des.max-open-files=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/share/elasticsearch/logs/heapdump.hprof -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.3.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ -Des.pidfile=/var/run/elasticsearch/10.37.38.124-data1/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/usr/local/var/log/elasticsearch/10.37.38.124-data1 -Des.default.path.data=/app/data/elasticsearch/10.37.38.124-data1 -Des.default.path.conf=/etc/elasticsearch/data1 org.elasticsearch.bootstrap.Elasticsearch ES\_HEAP\_SIZE=30g MAX\_LOCKED\_MEMORY=unlimited # Additional Java OPTS # es\_java\_opts: "$ES\_JAVA\_OPTS -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port= -Des.max-open-files=true", ES\_GC\_OPTS="-XX:+UseG1GC -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+UseCompressedOops -XX:MaxGCPauseMillis=200 -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=20 -XX:+UseStringDeduplication -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3335 -Des.max-open-files=true" export ES\_GC\_OPTS Step-3: Please find the settings. action.auto\_create\_index: true action.destructive\_requires\_name: true action.disable\_delete\_all\_indices: true bootstrap.mlockall: true cluster.name: Cluster\_name cluster.routing.allocation.same\_shard.host: true discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: master\_node1:9301,master\_node2:9301,master\_node3:9301 http.port: 9202 index.mapper.dynamic: true index.merge.policy.use\_compound\_file: false index.number\_of\_replicas: 0 index.number\_of\_shards: 96 index.query.bool.max\_clause\_count: 10000 index.refresh\_interval: 1000s indices.fielddata.cache.size: 10% indices.recovery.max\_bytes\_per\_sec: 60mb network.host: 0.0.0.0 node.data: true node.master: false script.inline: false script.stored: false script.file: false script.groovy.sandbox.enabled: false threadpool.bulk.queue\_size: 300 threadpool.index.queue\_size: 300 transport.tcp.port: 9302 threadpool.bulk.size: 60 threadpool.bulk.type: fixed threadpool.index.size: 60 threadpool.index.type: fixed threadpool.search.queue\_size: 400 threadpool.search.size: 60 threadpool.search.type: fixed discovery.zen.fd.ping\_timeout: 180s discovery.zen.fd.ping\_interval: 60s discovery.zen.fd.ping\_retries: 3 indices.cluster.send\_refresh\_mapping: false index.merge.policy.max\_merge\_at\_once: 10 index.merge.policy.reclaim\_deletes\_weight: 2.0 index.merge.policy.max\_merged\_segment: 5GB index.merge.policy.expunge\_deletes\_allowed: 10 index.merge.policy.segments\_per\_tier: 10 Step-4: Limits configuration setting under /etc/security/limits.conf # End of file # End of file # End of file - ``` soft nproc 65535 ``` - ``` hard nproc 65535 ``` - ``` soft nofile 65535 ``` - ``` hard nofile 65535 ``` elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited elasticsearch soft nproc 65535 elasticsearch hard nproc 65535 elasticsearch soft nofile 65535 elasticsearch hard nofile 65535 app soft nofile 16384 app hard nofile 16384 We have checked sestatus its already disabled in all the nodes (Centos7 we are using). sestatus SELinux status: disabled Free-memory: free -g total used free shared buff/cache available Mem: 503 91 410 0 1 410 Swap: 0 0 0 and also we are dropping the caches for every 5 mints. #Drop the page cache \*/5 \* \* \* \* sync; echo 1 \> /proc/sys/vm/drop\_caches We implemented above all the steps but still data nodes are using 24 to 25GB out of 30GB (90%) every time and its not releasing the GC and cluster become red and nodes are going down. Please suggest me anything we missed setting and configurations to fix this heap issue. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 6, 2018, 7:22pm UTC](https://discuss.elastic.co/t/elastic-search-heap-issues/159790/2 "2018-12-06T19:22:58Z")
Even though that is a very old version, I believe a lot in [this webinar](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch) should still be applicable. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2019, 7:23pm UTC](https://discuss.elastic.co/t/elastic-search-heap-issues/159790/3 "2019-01-03T19:23:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.