# ElasticSearch throwing “OutOfMemoryError\[unable to create new native thread\]” error **URL:** \ **Category:** Elasticsearch\ **Created:** [October 6, 2015, 10:35am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686 "2015-10-06T10:35:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 6, 2015, 10:35am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/1 "2015-10-06T10:35:26Z")
Hello, We have setup a cluster on the 2 BFM servers, I have only the basic cluster running with 1 data node, 1 master node and 1 client node. The master nodes will act as master + data and are configured to have a reserved JAVA heap size of each node is having 16 GB. Our server are highly configured along with 256 GB RAM. Pretty frequently, I notice that the data only node will fail due to out of memory error while searching. But when it fails, I see the error: [2015-10-05 04:43:11,731][INFO][http] [DEV\_DATA] bound\_address {inet[/0:0:0:0:0:0:0:0:9240]}, publish\_address {inet[/localhost:9240]} [2015-10-05 04:43:11,734]**[INFO][node] [DEV\_DATA] started** **[2015-10-05 04:44:26,566][ERROR][marvel.agent.exporter] [DEV\_DATA] create failure (index:[.marvel-2015.10.05] type: [node\_stats]): RemoteTransportException[[DEV\_MASTER][inet[/localhost:9300]][indices:data/write/bulk[s]]]; nested: OutOfMemoryError[unable to create new native thread];** [2015-10-05 04:44:58,179][WARN][indices.cluster] [DEV\_DATA] [[.marvel-2015.10.05][0]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [.marvel-2015.10.05][0]: Recovery failed from [DEV\_MASTER][F\_H\_SMI5T4imEDleZ4FZxg][[dayrhebfmd001.enterprisenet.org](http://dayrhebfmd001.enterprisenet.org)][inet[/localhost:9300]]{max\_local\_storage\_nodes=1, master=true} into [DEV\_DATA][muhubm9FSsevgdnyQJTb0Q][[dayrhebfmd001.enterprisenet.org](http://dayrhebfmd001.enterprisenet.org)][inet[[dayrhebfmd001.enterprisenet.org/localhost:9260](http://dayrhebfmd001.enterprisenet.org/localhost:9260)]]{max\_local\_storage\_nodes=1, master=false} at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280) at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70) at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) **Caused by: org.elasticsearch.transport.RemoteTransportException: [DEV\_MASTER][inet[/localhost:9300]][internal:index/shard/recovery/start\_recovery]** **Caused by: java.lang.OutOfMemoryError: unable to create new native thread** at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368) See the below screenshot of log details of data node and **please guide us to resolve this issue as soon as possible.** ![](https://us1.discourse-cdn.com/elastic/original/2X/1/14cf84d0f2691d60127fe5934eefda71465edec2.png) Thanks, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 6, 2015, 11:10am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/2 "2015-10-06T11:10:32Z")
When ES runs out of heap memory the problem is typically that you have reached the capacity of the cluster (or a single node), i.e. you have too much data or post too many queries for the hardware you have. Start by bumping the Elasticsearch's JVM heap. If you have 256 GB RAM you shouldn't have any issues increasing it from 16 to 31 GB (the limit before pointers become uncompressed). General methods to decrease heap usage include enabling doc values (default as of ES 2.0) and configuring the field data cache to actually start evicting fields once the cache is full. ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [October 8, 2015, 5:57pm UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/3 "2015-10-08T17:57:17Z")
Generally, I agree to what @magnusbaeck comment. Just my 2 cents from managing es production cluster. Sometime increase heap is good but for commodity server, but if too much heap to the jvm will be slow too given gc activities. But this is also subject your cluster use cases. Another point is the recent es exposes a lot of interesting metrics to monitor heap usage and you want to read on that and measure node jvm heap. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 9, 2015, 6:36am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/4 "2015-10-09T06:36:04Z")
Thanks for your responses @magnusbaeck @Jason_Wee I made some changes in yml file **indices.fielddata.cache.size: 75%** **indices.breaker.fielddata.limit: 85%** I would like to know is this right way to **enable doc values** for the field mappings? **Example for Sample mappings:-** Will doc values be enabled for the field type **"string"**? "load\_item": { "mappings": { "item": { "properties": { "load\_NAN": { "type": "string", "fields": { "load\_NAN\_CONTAINS": { "type": "string", "index\_analyzer": "str\_index\_analyzer" **“doc\_values:”true”** }, "load\_NAN\_RAW": { "type": "string", "index": **"not\_analyzed"** **“doc\_values:”true”** }, "load\_NAN\_STARTS": { "type": "string", "index\_analyzer": "prefix-analyzer", "search\_analyzer": "keyword" **“doc\_values”:”true”** } } }, "load\_STRT\_DT": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" **“doc\_values:”true”** } } } }}} Please let us know if it's correct or not.. Regards, Ganesh ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 9, 2015, 6:45am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/5 "2015-10-09T06:45:40Z")
Doc values won't work for analyzed strings. Otherwise it looks okay. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 9, 2015, 6:54am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/6 "2015-10-09T06:54:25Z")
Thanks for your quick response.. Is there any property for **field data to disable** during **bulk load indexing**? and same property can be **enable** during **searching time**? I am asking this questions to avoid out of memory error during searching time... You can give any other suggestions.. Regards, Ganesh ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 9, 2015, 8:11am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/7 "2015-10-09T08:11:47Z")
Your heap is OK. The exception means that your OS can not create new native threads. The number of threads in a JVM is limited by the OS process stack, not the JVM heap, so the exception message might be a bit misleading. Anyway, it takes many many thousands of threads to create such an exception. This vast number is not manageable by a JVM or an operating system and will bring down a process to a halt. So you should inspect your cluster configuration if you have thread pools configured that are unlimited. This is a very bad idea and therefore not configured by default. Unlimited thread pools are a classic method to bring a cluster down. Set up reasonable limits for thread pools. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 9, 2015, 11:21am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/8 "2015-10-09T11:21:03Z")
Thank for your response @jprante I checked the thread pool limit for node wise.. I figured thread pool settings with using the curl commands **curl -XGET " localhost:9200/\_nodes/thread\_pool?pretty"** Questions:- 1. Is this unlimited threadpools setttings you are talking about ? 2. How I change the settings of thread pool size? To determine the size of thread pool you need to know # of processors I am having **48 cores of cpu** 3, what is ideal value for setting # of processors? 4 should I add these settings in .yml file or using curl command? Please let us know.. **Samples threadpool settings** "cluster\_name" : "ES\_DEV", "nodes" : { "ycvddsfsdAD221cww" : { "name" : "Server\_DEV\_DATA", "transport\_address" : "inet[/localhost:9200]", "host" : "localhost.enterprisenet.org", "ip" : "localhost", "version" : "1.7.2", "build" : "e43676b", "http\_address" : "inet[/localhost:9200]", "attributes" : { "max\_local\_storage\_nodes" : "1", "master" : "false" }, "thread\_pool" : { "index" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "200" }, "search" : { "type" : "fixed", "min" : 49, "max" : 49, "queue\_size" : "1k" }, "bulk" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "50" } } }, "y8ilzvjfR0abu0L8Yg" : { "name" : "DEV\_CLIENT", "transport\_address" : "inet[/localhost:9200]", "host" : " localhost.enterprisenet.org", "ip" : " localhost", "version" : "1.7.2", "http\_address" : "inet[/localhost:9200]", "attributes" : { "max\_local\_storage\_nodes" : "1", "data" : "false", "master" : "false" }, "thread\_pool" : { "index" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "200" }, "search" : { "type" : "fixed", "min" : 49, "max" : 49, "queue\_size" : "1k" }, "bulk" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "50" } } } }, "KxAv6KaEfBaLcsz63g" : { "name" : "DEV\_MASTER", "transport\_address" : "inet[localhost.enterprisenet.org/ localhost:9200]", "host" : " localhost:9200.enterprisenet.org", "ip" : " localhost", "version" : "1.7.2", "http\_address" : "inet[/localhost:9200]", "attributes" : { "max\_local\_storage\_nodes" : "1", "master" : "true" }, "thread\_pool" : { "index" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "200" }, "search" : { "type" : "fixed", "min" : 49, "max" : 49, "queue\_size" : "1k" }, "bulk" : { "type" : "fixed", "min" : 32, "max" : 32, "queue\_size" : "50" } } } } } Regards, Ganeshbabu R ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 9, 2015, 11:36am UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/9 "2015-10-09T11:36:38Z")
Hi Magnus, For my understanding I want some clarification regarding analyzed strings **Examples of analyzed strings** The fields type and having analyzer "fields": { "load\_NAN\_CONTAINS": { "type": "string", "index\_analyzer": "str\_index\_analyzer" “doc\_values:”true” }, **Examples of not\_analyzed strings** "load\_NAN\_RAW": { "type": "string", **"index": "not\_analyzed"** **“doc\_values:”true”** }, **Examples of type strings** "IMAGE\_IND": { "type": "string", **“doc\_values:”true”** }, "HIST\_IND": { "type": "string", **“doc\_values:”true”** }, Please let me know its correct or not.. Thanks, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 9, 2015, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/10 "2015-10-09T12:05:23Z")
I _think_ `index` defaults to `analyzed`, in which case IMAGE\_IND and HIST\_IND are analyzed and can't use doc values. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [October 9, 2015, 1:03pm UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/11 "2015-10-09T13:03:35Z")
Thanks for your response. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:45pm UTC](https://discuss.elastic.co/t/elasticsearch-throwing-outofmemoryerror-unable-to-create-new-native-thread-error/31686/12 "2017-07-05T23:45:39Z")