# Java OOM error on a 64G - 8 Core CPU server **URL:** https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617 **Category:** Elasticsearch **Created:** [September 16, 2013, 9:51am UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617 "2013-09-16T09:51:54Z") **Posts on this page:** 6 **Page:** 1
### Author: ![11128](https://avatars.discourse-cdn.com/v4/letter/1/d07c76/32.png) [@11128](https://discuss.elastic.co/u/11128) #### Post date: [September 16, 2013, 9:51am UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/1 "2013-09-16T09:51:54Z")
Hello, Gurus I know this must be an old question. I have a ES instance\*(Kibana+ES+Fluentd as log analyse system, indexing nginx access log) \*running on 64G - 8 core CPU VM with DEFAULT configuration. We create a index every, around 2G size. After a week, the ES stopped indexing after I fired a query. The error log said lucene fired OOM ☹ Seemed all 5 shards are marked as dead. As I restarted the ES this morning, i works well again, currently the JVM status is "jvm":{"timestamp":1379324673375,"uptime":"6 hours, 33 minutes, 59 seconds and 453 milliseconds","uptime\_in\_millis":23639453, "mem":{\* "heap\_used":"478.2mb","heap\_used\_in\_bytes":501468512,"heap\_committed":"759.5mb","heap\_committed\_in\_bytes":796393472,"non\_heap\_used":"41.5mb","non\_heap\_used\_in\_bytes":43601016,"non\_heap\_committed":"63.8mb","non\_heap\_committed\_in\_bytes":66994176, \* "pools":{"Code Cache":{"used":"8.4mb","used\_in\_bytes":8910400,"max":"48mb","max\_in\_bytes":50331648,"peak\_used":"8.5mb","peak\_used\_in\_bytes":8931904,"peak\_max":"48mb","peak\_max\_in\_bytes":50331648},"Par Eden Space":{"used":"20mb","used\_in\_bytes":21053344,"max":"273mb","max\_in\_bytes":286326784,"peak\_used":"68.3mb","peak\_used\_in\_bytes":71630848,"peak\_max":"273mb","peak\_max\_in\_bytes":286326784},"Par Survivor Space":{"used":"5.1mb","used\_in\_bytes":5382312,"max":"34.1mb","max\_in\_bytes":35782656,"peak\_used":"8.5mb","peak\_used\_in\_bytes":8912896,"peak\_max":"34.1mb","peak\_max\_in\_bytes":35782656},"CMS Old Gen":{"used":"453mb","used\_in\_bytes":475032856,"max":"682.6mb","max\_in\_bytes":715849728,"peak\_used":"513.1mb","peak\_used\_in\_bytes":538113400,"peak\_max":"682.6mb","peak\_max\_in\_bytes":715849728},"CMS Perm Gen":{"used":"33mb","used\_in\_bytes":34690616,"max":"82mb","max\_in\_bytes":85983232,"peak\_used":"33mb","peak\_used\_in\_bytes":34690616,"peak\_max":"82mb","peak\_max\_in\_bytes":85983232}}}, "threads":{"count":200,"peak\_count":203}, "gc":{"collection\_count":3796,"collection\_time":"16 seconds and 862 milliseconds","collection\_time\_in\_millis":16862,"collectors":{"ParNew":{"collection\_count":3755,"collection\_time":"15 seconds and 679 milliseconds","collection\_time\_in\_millis":15679},"ConcurrentMarkSweep":{"collection\_count":41,"collection\_time":"1 second and 183 milliseconds","collection\_time\_in\_millis":1183}}}, "buffer\_pools":{"direct":{"count":242,"used":"46.6mb","used\_in\_bytes":48868573,"total\_capacity":"46.6mb","total\_capacity\_in\_bytes":48868573},"mapped":{"count":0,"used":"0b","used\_in\_bytes":0,"total\_capacity":"0b","total\_capacity\_in\_bytes":0}}} My Question is, giving JVM heap 32G RAM, could it be a good idea to solve this issue? simply modify "elasticsearch.yml" and the heap size option? JAVA LOG _[2013-09-15 10:34:12,028][DEBUG][action.search.type] [Atom Bob] [logstash-2013.09.14][2], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5a8f717c]_ _java.lang.OutOfMemoryError: Java heap space_ - ``` at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:55)* ``` - ``` at ``` org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:155) \* - ``` at ``` org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) \* - ``` at ``` org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:128)\* - ``` at ``` org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:60) \* - ``` at ``` org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:129) \* - ``` at ``` org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69) \* - ``` at ``` org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68) \* - ``` at ``` org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:615)\* - ``` at ``` org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162) \* - ``` at ``` org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)\* - ``` at ``` org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)\* - ``` at ``` org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)\* - ``` at ``` org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)\* - ``` at ``` org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)\* - ``` at ``` org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:243) \* - ``` at ``` org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:141) \* - ``` at ``` org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) \* - ``` at ``` org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:212) \* - ``` at ``` org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:199) \* - ``` at ``` org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:185) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) \* - ``` at java.lang.Thread.run(Thread.java:724)* ``` _[2013-09-15 10:34:12,028][DEBUG][action.search.type] [Atom Bob] [logstash-2013.09.14][2], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@58672471]_ _java.lang.OutOfMemoryError: Java heap space_ _[2013-09-15 10:34:09,061][WARN][index.engine.robin] [Atom Bob] [logstash-2013.09.15][0] failed engine_ _java.lang.OutOfMemoryError: Java heap space_ - ``` at ``` org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:51) \* - ``` at ``` org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:47) \* - ``` at ``` org.elasticsearch.index.translog.fs.FsTranslog.add(FsTranslog.java:333)\* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:473) \* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:365) \* - ``` at ``` org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:319) \* - ``` at ``` org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402) \* - ``` at ``` org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) \* - ``` at ``` org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521) \* - ``` at ``` org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) \* - ``` at java.lang.Thread.run(Thread.java:724)* ``` _[2013-09-15 10:35:05,835][WARN][http.netty] [Atom Bob] Caught exception while handling client http traffic, closing connection [id: 0x0e15a6b0, /127.0.0.1:15493 =\> /127.0.0.1:9200]_ _java.lang.OutOfMemoryError: Java heap space_ _[2013-09-15 10:35:05,832][DEBUG][action.bulk] [Atom Bob] [logstash-2013.09.15][3] failed to execute bulk item (index) index {[logstash-2013.09.15][fluentd][LGxqK8MfS66N5YvrmlOK4w], source[{"test":"Stack trace:","\_key":"error.web5","@timestamp":"2013-09-15T10:33:28+08:00"}]}_ _org.elasticsearch.index.engine.CreateFailedEngineException: [logstash-2013.09.15][3] Create failed for [fluentd#LGxqK8MfS66N5YvrmlOK4w]_ - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:378) \* - ``` at ``` org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:319) \* - ``` at ``` org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402) \* - ``` at ``` org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) \* - ``` at ``` org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521) \* - ``` at ``` org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) \* - ``` at ``` java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) \* - ``` at java.lang.Thread.run(Thread.java:724)* ``` _Caused by: org.apache.lucene.store.AlreadyClosedException: this ReferenceManager is closed_ - ``` at ``` org.apache.lucene.search.ReferenceManager.acquire(ReferenceManager.java:97)\* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.searcher(RobinEngine.java:739) \* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.loadCurrentVersionFromIndex(RobinEngine.java:1319) \* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.innerCreate(RobinEngine.java:391) \* - ``` at ``` org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:365) \* - ``` ... 8 more* ``` - - _......_ _Caused by: java.lang.OutOfMemoryError: Java heap space_ _[2013-09-15 11:03:52,847][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][2], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:52,848][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][2], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:52,850][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][0], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:52,850][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][0], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:53,387][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][4], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:53,387][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][4], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:53,389][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][1], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:53,389][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][1], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]_ _[2013-09-15 11:03:53,398][WARN][indices.cluster] [Atom Bob] [logstash-2013.09.15][0] master [[Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]]] marked shard as started, but shard has not been created, mark shard as failed_ _[2013-09-15 11:03:53,399][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][0], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [master [Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]] marked shard as started, but shard has not been created, mark shard as failed]_ _[2013-09-15 11:03:53,399][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][0], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [master [Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]] marked shard as started, but shard has not been created, mark shard as failed]_ _[2013-09-15 11:03:53,399][WARN][indices.cluster] [Atom Bob] [logstash-2013.09.15][1] master [[Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]]] marked shard as started, but shard has not been created, mark shard as failed_ _[2013-09-15 11:03:53,400][WARN][cluster.action.shard] [Atom Bob] sending failed shard for [logstash-2013.09.15][1], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [master [Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]] marked shard as started, but shard has not been created, mark shard as failed]_ \*\* _[2013-09-15 11:03:53,400][WARN][cluster.action.shard] [Atom Bob] received shard failed for [logstash-2013.09.15][1], node[uT\_403fVTT-oI524d7hK1Q], [P], s[STARTED], reason [master [Atom Bob][uT\_403fVTT-oI524d7hK1Q][inet[/10.50.1.79:9300]] marked shard as started, but shard has not been created, mark shard as failed]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
### Author: ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr) #### Post date: [September 16, 2013, 1:59pm UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/2 "2013-09-16T13:59:52Z")
Xie Lebing wrote: > My Question is, giving JVM heap 32G RAM, could it be a good idea > to solve this issue? simply modify "elasticsearch.yml" and the > heap size option? You indeed want to set the heap to something, and half the machine RAM is a good starting point. In your case though I would set it 31g because the JVM will uncompress object pointers starting around 32g and you'll actually lose space. Use the environment variable ES\_HEAP\_SIZE in whatever process starts Elasticsearch. ES\_HEAP\_SIZE=31g bin/elasticsearch ... Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
### Author: ![11128](https://avatars.discourse-cdn.com/v4/letter/1/d07c76/32.png) [@11128](https://discuss.elastic.co/u/11128) #### Post date: [September 22, 2013, 4:58am UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/3 "2013-09-22T04:58:51Z")
Thank you for reply. with the default heap option (1G), I hit the OOM error again and JVM info looks like: {"cluster\_name":"elasticsearch","nodes":{"rG-uC\_WtS0yA2oER-hSv3w":{"timestamp":1379823196938,"name":"Iron Maiden","transport\_address":"inet[/XXXXXXX:9300]","hostname":"XXXXX","jvm":{"timestamp":1379823196938,"uptime":"145 hours, 2 minutes, 43 seconds and 16 milliseconds","uptime\_in\_millis":522163016,"mem\*":{"heap\_used":"959.6mb","heap\_used\_in\_bytes":1006288952,"heap\_committed":"989.8mb","heap\_committed\_in\_bytes":1037959168,"non\_heap\_used":"43.3mb","non\_heap\_used\_in\_bytes":45420880,"non\_heap\_committed":"66.2mb","non\_heap\_committed\_in\_bytes":69423104,"pools":{"Code Cache":{"used":"9.7mb","used\_in\_bytes":10245568,"max":"48mb","max\_in\_bytes":50331648,"peak\_used":"9.8mb","peak\_used\_in\_bytes":10361792,"peak\_max":"48mb","peak\_max\_in\_bytes":50331648}, \*"Par Eden Space":{"used":"273mb","used\_in\_bytes":286326776,"max":"273mb","max\_in\_bytes":286326784,"peak\_used":"273mb","peak\_used\_in\_bytes":286326784,"peak\_max":"273mb","peak\_max\_in\_bytes":286326784},"Par Survivor Space":{"used":"4.3mb","used\_in\_bytes":4582320,"max":"34.1mb","max\_in\_bytes":35782656,"peak\_used":"34.1mb","peak\_used\_in\_bytes":35782656,"peak\_max":"34.1mb","peak\_max\_in\_bytes":35782656},"CMS Old Gen":{"used":"682.2mb","used\_in\_bytes":715380344,"max":"682.6mb","max\_in\_bytes":715849728,"peak\_used":"682.6mb","peak\_used\_in\_bytes":715849728,"peak\_max":"682.6mb","peak\_max\_in\_bytes":715849728},"CMS Perm Gen":{"used":"33.5mb","used\_in\_bytes":35175312,"max":"82mb","max\_in\_bytes":85983232,"peak\_used":"33.5mb","peak\_used\_in\_bytes":35181568,"peak\_max":"82mb","peak\_max\_in\_bytes":85983232}}},"threads":{"count":186,"peak\_count":217},"gc":{"collection\_count":158700,"collection\_time":"10 hours, 31 minutes, 2 seconds and 255 milliseconds","collection\_time\_in\_millis":37862255,"collectors":{"ParNew":{"collection\_count":43964,"collection\_time":"3 minutes, 11 seconds and 945 milliseconds","collection\_time\_in\_millis":191945},"ConcurrentMarkSweep":{"collection\_count":114736,"collection\_time":"10 hours, 27 minutes, 50 seconds and 310 milliseconds","collection\_time\_in\_millis":37670310}}},"buffer\_pools":{"direct":{"count":245,"used":"70.7mb","used\_in\_bytes":74193116,"total\_capacity":"70.7mb","total\_capacity\_in\_bytes":74193116},"mapped":{"count":0,"used":"0b","used\_in\_bytes":0,"total\_capacity":"0b","total\_capacity\_in\_bytes":0}}}}}} Set the heap size to 20-30G could solve it right? Thanks 2013/9/16 Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) > Xie Lebing wrote: > > My Question is, giving JVM heap 32G RAM, could it be a good idea to solve > > > this issue? simply modify "elasticsearch.yml" and the heap size option? > > You indeed want to set the heap to something, and half the machine RAM is > a good starting point. In your case though I would set it 31g because the > JVM will uncompress object pointers starting around 32g and you'll actually > lose space. Use the environment variable ES\_HEAP\_SIZE in whatever process > starts Elasticsearch. > > ES\_HEAP\_SIZE=31g bin/elasticsearch ... > > Drew > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- 发自我的 iSpaceship [http://cn.linkedin.com/in/xielebing](http://cn.linkedin.com/in/xielebing) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
### Author: ![11128](https://avatars.discourse-cdn.com/v4/letter/1/d07c76/32.png) [@11128](https://discuss.elastic.co/u/11128) #### Post date: [September 22, 2013, 3:58pm UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/4 "2013-09-22T15:58:13Z")
Guys, I decided to extend Heap size to 4G and modify the Java Option like _JAVA\_OPTS="$JAVA\_OPTS -server **-Xms4096m -Xmx4096g -Xmn2048m** -Djava.awt.headless=true -XX:PermSize=256m -XX:MaxPermSize=256m -XX:ParallelGCThreads=8 -XX:SurvivorRatio=1_ _8 -Xnoclassgc -XX:MaxTenuringThreshold=10 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompa_ _ction=5 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintGCDetails -XX:+PrintG_ _CTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/var/elasticsearch -cp :/var/elasticsearch/lib/elasticsearch-0.90.3.jar:_ _/var/elasticsearch/lib/_:/var/elasticsearch/lib/sigar/\* org.elasticsearch.bootstrap.Elasticsearch"\* After I restart the ES, the server crashed. See attched top. Seemed Java consumed all the memory. Any tips about this issue? thanks you! Sep 22 19:45:13 srv-log3 kernel: INFO: task java:6634 blocked for more than 120 seconds. Sep 22 19:45:13 srv-log3 kernel: "echo 0 \> /proc/sys/kernel/hung\_task\_timeout\_secs" disables this message. Sep 22 19:45:13 srv-log3 kernel: java D ffffffff80157f06 0 6634 1 6635 6633 (NOTLB) Sep 22 19:45:13 srv-log3 kernel: ffff810802407e18 0000000000000082 0000000000000001 000000000b398800 Sep 22 19:45:13 srv-log3 kernel: 00000000ffffffda 0000000000000007 ffff81082df38830 ffff81102a7947f0 Sep 22 19:45:13 srv-log3 kernel: 000002f272b170c3 0000000000009da0 ffff81082df38a18 0000000200000000 Sep 22 19:45:13 srv-log3 kernel: Call Trace: Sep 22 19:45:13 srv-log3 kernel: [] \_\_down\_read+0x7a/0x92 Sep 22 19:45:13 srv-log3 kernel: [] do\_page\_fault+0x414/0x842 Sep 22 19:45:13 srv-log3 kernel: [] thread\_return+0x62/0xfe Sep 22 19:45:13 srv-log3 kernel: [] error\_exit+0x0/0x84 On Sunday, September 22, 2013 12:58:51 PM UTC+8, Xie Lebing wrote: > Thank you for reply. with the default heap option (1G), I hit the OOM > error again and JVM info looks like: > > {"cluster\_name":"elasticsearch","nodes":{"rG-uC\_WtS0yA2oER-hSv3w":{"timestamp":1379823196938,"name":"Iron > Maiden","transport\_address":"inet[/XXXXXXX:9300]","hostname":"XXXXX","jvm":{"timestamp":1379823196938,"uptime":"145 > hours, 2 minutes, 43 seconds and 16 > milliseconds","uptime\_in\_millis":522163016,"mem\*":{"heap\_used":"959.6mb","heap\_used\_in\_bytes":1006288952,"heap\_committed":"989.8mb","heap\_committed\_in\_bytes":1037959168,"non\_heap\_used":"43.3mb","non\_heap\_used\_in\_bytes":45420880,"non\_heap\_committed":"66.2mb","non\_heap\_committed\_in\_bytes":69423104,"pools":{"Code > Cache":{"used":"9.7mb","used\_in\_bytes":10245568,"max":"48mb","max\_in\_bytes":50331648,"peak\_used":"9.8mb","peak\_used\_in\_bytes":10361792,"peak\_max":"48mb","peak\_max\_in\_bytes":50331648}, > \*"Par Eden > Space":{"used":"273mb","used\_in\_bytes":286326776,"max":"273mb","max\_in\_bytes":286326784,"peak\_used":"273mb","peak\_used\_in\_bytes":286326784,"peak\_max":"273mb","peak\_max\_in\_bytes":286326784},"Par > Survivor > Space":{"used":"4.3mb","used\_in\_bytes":4582320,"max":"34.1mb","max\_in\_bytes":35782656,"peak\_used":"34.1mb","peak\_used\_in\_bytes":35782656,"peak\_max":"34.1mb","peak\_max\_in\_bytes":35782656},"CMS > Old > Gen":{"used":"682.2mb","used\_in\_bytes":715380344,"max":"682.6mb","max\_in\_bytes":715849728,"peak\_used":"682.6mb","peak\_used\_in\_bytes":715849728,"peak\_max":"682.6mb","peak\_max\_in\_bytes":715849728},"CMS > Perm > Gen":{"used":"33.5mb","used\_in\_bytes":35175312,"max":"82mb","max\_in\_bytes":85983232,"peak\_used":"33.5mb","peak\_used\_in\_bytes":35181568,"peak\_max":"82mb","peak\_max\_in\_bytes":85983232}}},"threads":{"count":186,"peak\_count":217},"gc":{"collection\_count":158700,"collection\_time":"10 > hours, 31 minutes, 2 seconds and 255 > milliseconds","collection\_time\_in\_millis":37862255,"collectors":{"ParNew":{"collection\_count":43964,"collection\_time":"3 > minutes, 11 seconds and 945 > milliseconds","collection\_time\_in\_millis":191945},"ConcurrentMarkSweep":{"collection\_count":114736,"collection\_time":"10 > hours, 27 minutes, 50 seconds and 310 > milliseconds","collection\_time\_in\_millis":37670310}}},"buffer\_pools":{"direct":{"count":245,"used":"70.7mb","used\_in\_bytes":74193116,"total\_capacity":"70.7mb","total\_capacity\_in\_bytes":74193116},"mapped":{"count":0,"used":"0b","used\_in\_bytes":0,"total\_capacity":"0b","total\_capacity\_in\_bytes":0}}}}}} > > Set the heap size to 20-30G could solve it right? Thanks > > 2013/9/16 Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) > > > Xie Lebing wrote: > > > > My Question is, giving JVM heap 32G RAM, could it be a good idea to > > > > > solve this issue? simply modify "elasticsearch.yml" and the heap size > > > option? > > > > You indeed want to set the heap to something, and half the machine RAM is > > a good starting point. In your case though I would set it 31g because the > > JVM will uncompress object pointers starting around 32g and you'll actually > > lose space. Use the environment variable ES\_HEAP\_SIZE in whatever process > > starts Elasticsearch. > > > > ES\_HEAP\_SIZE=31g bin/elasticsearch ... > > > > Drew > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > 发自我的 iSpaceship > [http://cn.linkedin.com/in/xielebing](http://cn.linkedin.com/in/xielebing) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
### Author: ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante) #### Post date: [September 23, 2013, 7:21am UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/5 "2013-09-23T07:21:08Z")
Please look more carefully at your options to follow Drew's suggestions. - ES\_HEAP\_SIZE=31g, this sets -Xmx31g -Xms31g (and I recommend mmapfs and bootstrap mlockall in this case) - do not set -Xmn - do not set -Xmx4096g unless you have 8 terabytes of memory Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/java-oom-error-on-a-64g-8-core-cpu-server/13617/6 "2017-07-06T02:15:10Z")