# Exiting java.lang.OutOfMemoryError: Java heap space **URL:** \ **Category:** Elasticsearch\ **Created:** [February 4, 2019, 9:44am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926 "2019-02-04T09:44:28Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![53dd57e6c0d57177c379](https://avatars.discourse-cdn.com/v4/letter/5/48db29/32.png) [@53dd57e6c0d57177c379](https://discuss.elastic.co/u/53dd57e6c0d57177c379)\ **Post date:** [February 4, 2019, 9:44am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/1 "2019-02-04T09:44:28Z")
[2019-02-04T12:33:54,521][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] fatal error in thread [elasticsearch[8mTx-ra][search][T#7]], exiting java.lang.OutOfMemoryError: Java heap space Please help me solve this issue.. elasticsearch-5.6.14 What i try to do .. 1. On sysconfig: /etc/sysconfig/elasticsearch you should have: ES\_JAVA\_OPTS="-Xms3g -Xmx3g" MAX\_LOCKED\_MEMORY=unlimited 2. /etc/security/limits.conf On security limits config: /etc/security/limits.conf you should have elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited 3. On the service script: /usr/lib/systemd/system/elasticsearch.service you should uncomment: LimitMEMLOCK=infinity you should do systemctl daemon-reload after changing the service script 4. /etc/elasticsearch/elasticsearch.yml bootstrap.memory\_lock: true Zero result. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 9:58am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/2 "2019-02-04T09:58:55Z")
What is the output of the [cluster health](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-health.html) and [cluster stats APIs](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-stats.html)? ---
**Author:** ![53dd57e6c0d57177c379](https://avatars.discourse-cdn.com/v4/letter/5/48db29/32.png) [@53dd57e6c0d57177c379](https://discuss.elastic.co/u/53dd57e6c0d57177c379)\ **Post date:** [February 4, 2019, 11:19am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/3 "2019-02-04T11:19:03Z")
curl -X GET "localhost:9200/\_cluster/health" {"cluster\_name":"graylog","status":"green","timed\_out":false,"number\_of\_nodes":1,"number\_of\_data\_nodes":1,"active\_primary\_shards":24,"active\_shards":24,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":100.0} { "\_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster\_name" : "graylog", "timestamp" : 1549278763118, "status" : "green", "indices" : { "count" : 6, "shards" : { "total" : 24, "primaries" : 24, "replication" : 0.0, "index" : { "shards" : { "min" : 4, "max" : 4, "avg" : 4.0 }, "primaries" : { "min" : 4, "max" : 4, "avg" : 4.0 }, "replication" : { "min" : 0.0, "max" : 0.0, "avg" : 0.0 } } }, "docs" : { "count" : 105302466, "deleted" : 0 }, "store" : { "size" : "36.7gb", "size\_in\_bytes" : 39417555657, "throttle\_time" : "0s", "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size" : "7.7mb", "memory\_size\_in\_bytes" : 8142541, "total\_count" : 5240, "hit\_count" : 2596, "miss\_count" : 2644, "cache\_size" : 79, "cache\_count" : 79, "evictions" : 0 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 87, "memory" : "72.1mb", "memory\_in\_bytes" : 75602569, "terms\_memory" : "53.8mb", "terms\_memory\_in\_bytes" : 56444921, "stored\_fields\_memory" : "12.7mb", "stored\_fields\_memory\_in\_bytes" : 13319136, "term\_vectors\_memory" : "0b", "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory" : "16.3kb", "norms\_memory\_in\_bytes" : 16704, "points\_memory" : "4.7mb", "points\_memory\_in\_bytes" : 5003684, "doc\_values\_memory" : "798.9kb", "doc\_values\_memory\_in\_bytes" : 818124, "index\_writer\_memory" : "7.4mb", "index\_writer\_memory\_in\_bytes" : 7858796, "version\_map\_memory" : "12.2kb", "version\_map\_memory\_in\_bytes" : 12584, "fixed\_bit\_set" : "0b", "fixed\_bit\_set\_memory\_in\_bytes" : 0, "max\_unsafe\_auto\_id\_timestamp" : -1, "file\_sizes" : { } } }, "nodes" : { "count" : { "total" : 1, "data" : 1, "coordinating\_only" : 0, "master" : 1, "ingest" : 1 }, "versions" : [ "5.6.14" ], "os" : { "available\_processors" : 8, "allocated\_processors" : 8, "names" : [ { "name" : "Linux", "count" : 1 } ], "mem" : { "total" : "15.5gb", "total\_in\_bytes" : 16646242304, "free" : "606.5mb", "free\_in\_bytes" : 636063744, "used" : "14.9gb", "used\_in\_bytes" : 16010178560, "free\_percent" : 4, "used\_percent" : 96 } }, "process" : { "cpu" : { "percent" : 1 }, "open\_file\_descriptors" : { "min" : 320, "max" : 320, "avg" : 320 } }, "jvm" : { "max\_uptime" : "1.6h", "max\_uptime\_in\_millis" : 5862002, "versions" : [ { "version" : "1.8.0\_191", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.191-b12", "vm\_vendor" : "Oracle Corporation", "count" : 1 } ], "mem" : { "heap\_used" : "1.3gb", "heap\_used\_in\_bytes" : 1443017576, "heap\_max" : "2.9gb", "heap\_max\_in\_bytes" : 3151495168 }, "threads" : 62 }, "fs" : { "total" : "397gb", "total\_in\_bytes" : 426350342144, "free" : "343.1gb", "free\_in\_bytes" : 368414822400, "available" : "343.1gb", "available\_in\_bytes" : 368414822400, "spins" : "true" }, "plugins" : , "network\_types" : { "transport\_types" : { "netty4" : 1 }, "http\_types" : { "netty4" : 1 } } } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 11:23am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/4 "2019-02-04T11:23:43Z")
That looks fine. Wonder if indexing and querying require more heap. Have you tried increasing it beyond 3GB? ---
**Author:** ![53dd57e6c0d57177c379](https://avatars.discourse-cdn.com/v4/letter/5/48db29/32.png) [@53dd57e6c0d57177c379](https://discuss.elastic.co/u/53dd57e6c0d57177c379)\ **Post date:** [February 4, 2019, 11:39am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/5 "2019-02-04T11:39:46Z")
Of course, i tried set heap size 7G but is do not have any result.. cant understand why elastic service down. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 11:42am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/6 "2019-02-04T11:42:01Z")
Is there anything in the logs? ---
**Author:** ![53dd57e6c0d57177c379](https://avatars.discourse-cdn.com/v4/letter/5/48db29/32.png) [@53dd57e6c0d57177c379](https://discuss.elastic.co/u/53dd57e6c0d57177c379)\ **Post date:** [February 4, 2019, 11:50am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/7 "2019-02-04T11:50:56Z")
```auto [2019-02-04T12:26:51,415][INFO][o.e.n.Node] initialized [2019-02-04T12:26:51,415][INFO][o.e.n.Node] [8mTx-ra] starting ... [2019-02-04T12:26:51,562][INFO][o.e.t.TransportService] [8mTx-ra] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2019-02-04T12:26:54,607][INFO][o.e.c.s.ClusterService] [8mTx-ra] new_master {8mTx-ra}{8mTx-raJQ0yYHnDuKXiFzg}{bZtINvTZSPOWlAaiNniWXg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2019-02-04T12:26:54,619][INFO][o.e.h.n.Netty4HttpServerTransport] [8mTx-ra] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2019-02-04T12:26:54,620][INFO][o.e.n.Node] [8mTx-ra] started [2019-02-04T12:26:54,864][INFO][o.e.g.GatewayService] [8mTx-ra] recovered [6] indices into cluster_state [2019-02-04T12:26:55,919][INFO][o.e.c.r.a.AllocationService] [8mTx-ra] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][2], [graylog_0][0]] ...]). [2019-02-04T12:33:33,974][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][402] overhead, spent [760ms] collecting in the last [1.4s] [2019-02-04T12:33:34,974][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][403] overhead, spent [718ms] collecting in the last [1s] [2019-02-04T12:33:36,081][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][404] overhead, spent [818ms] collecting in the last [1.1s] [2019-02-04T12:33:37,155][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][405] overhead, spent [721ms] collecting in the last [1s] [2019-02-04T12:33:38,845][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][406] overhead, spent [1.6s] collecting in the last [1.6s] [2019-02-04T12:33:40,028][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][407] overhead, spent [911ms] collecting in the last [1.1s] [2019-02-04T12:33:42,552][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][408] overhead, spent [2.2s] collecting in the last [2.5s] [2019-02-04T12:33:45,932][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][409] overhead, spent [3.3s] collecting in the last [3.3s] [2019-02-04T12:33:49,945][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][410] overhead, spent [4s] collecting in the last [4s] [2019-02-04T12:33:53,016][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][411] overhead, spent [3s] collecting in the last [3s] [2019-02-04T12:33:54,520][WARN][o.e.m.j.JvmGcMonitorService] [8mTx-ra] [gc][412] overhead, spent [1.4s] collecting in the last [1.5s] [2019-02-04T12:33:54,521][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[8mTx-ra][search][T#7]], exiting java.lang.OutOfMemoryError: Java heap space [2019-02-04T12:33:54,521][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[8mTx-ra][search][T#5]], exiting java.lang.OutOfMemoryError: Java heap space [2019-02-04T12:35:02,962][INFO][o.e.n.Node] [] initializing ... ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 11:53am UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/8 "2019-02-04T11:53:51Z")
What does your configuration look like? Do you have any non-default settings that could be causing problems? ---
**Author:** ![53dd57e6c0d57177c379](https://avatars.discourse-cdn.com/v4/letter/5/48db29/32.png) [@53dd57e6c0d57177c379](https://discuss.elastic.co/u/53dd57e6c0d57177c379)\ **Post date:** [February 4, 2019, 12:05pm UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/9 "2019-02-04T12:05:27Z")
Configuration of what? No, I do not think so. Is it so hard? to make the elastic not fall with a silly mistake? Why can't he refuse the request? Why is the service crashing? curl -X GET "localhost:9200/\_nodes?filter\_path=\*\*.mlockall" {"nodes":{"8mTx-raJQ0yYHnDuKXiFzg":{"process":{"mlockall":true}}}} ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 4, 2019, 6:15pm UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/10 "2019-02-04T18:15:29Z")
> [@53dd57e6c0d57177c379](#): > > Is it so hard? to make the elastic not fall with a silly mistake? Yes, it is in fact really rather hard to do this. We'll try and help, but you will have to help too. I've never seen a node go out of memory within 7 minutes of startup before, so there's definitely something unusual with your configuration or usage pattern. Perhaps you have found a bug. It'd be good to find out. When your node went down with an `OutOfMemoryError` it will (by default) write a heap dump. It would be very useful if you could share this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 4, 2019, 6:15pm UTC](https://discuss.elastic.co/t/exiting-java-lang-outofmemoryerror-java-heap-space/166926/11 "2019-03-04T18:15:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.