# Elasticsearch 5.5.0 cluster crash -- elasticsearch.yml processors set too high **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2017, 1:53pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397 "2017-07-09T13:53:20Z")\ **Posts on this page:** 8\ **Page:** 2
**Author:** ![LogBabel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/logbabel/32/45107_2.png) [@LogBabel](https://discuss.elastic.co/u/LogBabel)\ **Post date:** [July 11, 2017, 4:45pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/21 "2017-07-11T16:45:27Z")
The cluster has been stable since downgrading from Java8-131 to 112. # Cluster options [cluster.name](http://cluster.name): ES\_CLUSTER1 [node.name](http://node.name): "node1" node.attr.rack: "rackA6-1" node.master: true node.data: true # # Data options path.data: /opt/elasticsearch path.logs: /var/log/elasticsearch # # Memory options bootstrap.memory\_lock: true # # Network options network.host: "10.10.10.1" http.port: 9200 # # Discovery options discovery.zen.ping.unicast.hosts: ["10.10.10.1", "10.10.10.2", "10.10.10.3"] discovery.zen.minimum\_master\_nodes: 2 # # Gateway gateway.recover\_after\_nodes: 3 # # # Index cache settings indices.memory.index\_buffer\_size: 5% # # Other processors: 320 action.auto\_create\_index: .security,.monitoring\*,.watches,.triggered\_watches,.watcher-history\*,logstash-\* action.destructive\_requires\_name: true # # X-Pack settings xpack.ml.enabled: "false" xpack.security.enabled: "false" ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 11, 2017, 9:26pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/22 "2017-07-11T21:26:33Z")
Good to hear. Did you upgrade java at the same time as Elasticsearch, or before/after? ---
**Author:** ![pickypg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pickypg/32/62409_2.png) [@pickypg](https://discuss.elastic.co/u/pickypg)\ **Post date:** [July 12, 2017, 1:26am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/23 "2017-07-12T01:26:54Z")
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread ```auto processors: 320 ``` How many cores are on this machine? This is the root cause of the issue and I will be surprised if any version of Java is able to properly handle the number of native threads that you are implicitly trying to create. My guess is that you can simply unset that setting and things should go back to working. If this box actually has 320 logical cores available to it, then you should use containers or VMs to carve up the box into multiple nodes rather than dedicate 320 cores to a single node. The JVM is simply unable to sustain that. An example of [this type of failure can be seen here](https://github.com/elastic/elasticsearch/issues/3478). Fortunately the JVM has improved since then, but not to 320 cores (the default max is now the number of available processors, but it used to be 24 and then 32; by setting the value explicitly you control its value). ---
**Author:** ![LogBabel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/logbabel/32/45107_2.png) [@LogBabel](https://discuss.elastic.co/u/LogBabel)\ **Post date:** [July 12, 2017, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/24 "2017-07-12T06:19:09Z")
At the same time. Lesson learned. ---
**Author:** ![LogBabel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/logbabel/32/45107_2.png) [@LogBabel](https://discuss.elastic.co/u/LogBabel)\ **Post date:** [July 12, 2017, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/25 "2017-07-12T06:40:42Z")
I've used this setting for a long time now with previous versions. Maybe your right in that it causes problems with Java8-131, maybe because it should and is now enforced in some way. I guess I misread the Elasticsearch documentation in thinking more threads can boost performance on otherwise underutilized CPU resources, I believe it was recommended to try doubling the value or even more but I cannot find the reference now. The boxes have 20 core / 40 thread count. From trial and error I reached that value, I think it related to issues with index and the bulk\_queue pools. The default values would hardly reach 5k EPS and from trial and error I've been able to boost the performance up to 25k - 50k EPS. Maybe it's time to re-test and take better notes. There are a myriad of variables to track when tuning for performance. ---
**Author:** ![LogBabel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/logbabel/32/45107_2.png) [@LogBabel](https://discuss.elastic.co/u/LogBabel)\ **Post date:** [July 12, 2017, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/26 "2017-07-12T07:18:30Z")
I think you are correct.. two nodes just crashed with the same error and this is using Java8-112. ---
**Author:** ![LogBabel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/logbabel/32/45107_2.png) [@LogBabel](https://discuss.elastic.co/u/LogBabel)\ **Post date:** [July 12, 2017, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/27 "2017-07-12T15:42:00Z")
I changed the processors setting to 40 and the thread count dropped from ~1700 to ~500. Seems stable so far. Thanks for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 9, 2017, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397/28 "2017-08-09T15:42:21Z")
Closed on August 9, 2017, 3:42pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/elasticsearch-5-5-0-cluster-crash-elasticsearch-yml-processors-set-too-high/92397.md?page=1)