# Please suggest performance settings for my ElasticSearch cluster
**URL:** https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232
**Category:** Elasticsearch
**Created:** [September 11, 2016, 8:53am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232 "2016-09-11T08:53:13Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 8:53am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/1 "2016-09-11T08:53:13Z")
Hi, I have troubles with my ElasticSearch cluster and don't know what to do. Here my settings:
Server:
64GB of RAM
8 cores
500GB SSD
Data:
10 different index each 1M rows
10M of rows
11GB of data
Actual ES cluster
3 nodes on the same server
1 master node and 2 slave nodes
8g Heap size for each node
bootstrap.mlockall: true
Search requests:
Search Rate: 150 /s
Search Latency: 1ms
I have a lot of aggregations and filter for each search request
Problem:
When the JVM heap is coming over 90% the nodes not responding anymore. When I restart them, then everything works fine until the next 3 days, where the heap is coming again to 90% and the cluster not respond.
Here is the graph when the heap is \> 90%
[[Album] Imgur: The magic of the Internet  ](https://imgur.com/a/zi9XG)
What I need:
- Can someone suggest me settings for my elasticsearch.yml so I can handle in a good way the cache based on the settings above
- What to do with the \<90% JVM heap problem
Thanks
Nik
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2016, 9:13am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/2 "2016-09-11T09:13:12Z")
Why are you running multiple nodes on the server when you can simply run a single node with 30GB of heap?
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 9:15am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/3 "2016-09-11T09:15:32Z")
I try it, but the clean of the garbage collection has took longer and there were timeouts when the GC has start to clean.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2016, 9:27am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/4 "2016-09-11T09:27:03Z")
What does your node configuration look like? Do you have any custom settings or are you running with the defaults?
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 9:31am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/5 "2016-09-11T09:31:54Z")
This are my settings:
```
cluster.name: XXXX
node.name: XXX-master
node.data: true
node.master: true
bootstrap.mlockall: true
index.merge.policy.merge_factor: 5
# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 1000
threadpool.bulk.queue_size: 30000
# Index pool
threadpool.index.type: fixed
threadpool.index.size: 1000
threadpool.index.queue_size: 10000
#search pool
threadpool.search.queue_size: 4000
index.cache.query.enable: true
index.requests.cache.enable: true
indices.cache.query.size: 25%
indices.fielddata.cache.size: 25%
indices.cache.filter.size: 25%
transport.tcp.compress: true;
#index
index.store.type: mmapfs
network.bind_host: xxx.xxx.xxx.xxx
network.publish_host: xxx.xxx.xxx.xxx
network.host: xxx.xxx.xxx.xxx
discovery.zen.ping.unicast.hosts: ["xxx.xxx.xxx.xxx"]
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 10s
transport.tcp.port: 9300
http.port: 9200
http.max_content_length: 500mb
index.routing.allocation.disable_allocation: false
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
script.engine.groovy.inline.aggs: on
script.inline: on
script.indexed: on
index.max_result_window: 10000
```
When the JVM heap is high I start to become in the log files such messages:
`[2016-09-11 09:03:54,738][WARN][transport] [XXX-master-1] Transport response handler not found of id [16262715]`
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 11, 2016, 9:41am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/6 "2016-09-11T09:41:26Z")
> [@renego](#):
>
> # Bulk pool
>
> threadpool.bulk.type: fixed
> threadpool.bulk.size: 1000
> threadpool.bulk.queue\_size: 30000
>
> # Index pool
>
> threadpool.index.type: fixed
> threadpool.index.size: 1000
> threadpool.index.queue\_size: 10000
>
> #search pool
> threadpool.search.queue\_size: 4000
No no no no no.
All of this is in memory and having such ridiculously large settings is going to add _more_ heap pressure.
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 9:46am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/7 "2016-09-11T09:46:03Z")
To explain the process:
As I say I have 10 index, each one 1M rows. All this index has job ads for 10 different countries. 1 index one county. Every day once a day in different times I rebuild each index. This means I create a new one index for the selected country, fill the data into it and remove the old index.
Should I remove the settings above?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2016, 10:16am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/8 "2016-09-11T10:16:41Z")
> [@renego](#):
>
> Should I remove the settings above?
Yes. Start with the default settings and then start tweaking from that point if necessary. More is not always better when it comes to these settings and the defaults are generally very good in my experience.
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 10:47am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/9 "2016-09-11T10:47:16Z")
Ok I will remove them. What about the cache settings? Are they ok? What do you suggest?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2016, 12:52pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/10 "2016-09-11T12:52:01Z")
Unless you have reached these through systematic testing and evaluation, I would recommend starting with all default settings. Whether those cache settings are right or not for you use case is impossible for me to tell.
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 11, 2016, 4:32pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/11 "2016-09-11T16:32:19Z")
The question ist. I don't know it. Should the JVM heam reach 90% in good optimized node or not? What is a good avg % for the heap ?
Why my cluster die when the JVM heap is \> 90% I can not undestand the problem.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 13, 2016, 3:38pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/12 "2016-09-13T15:38:40Z")
Does it reach 90% after you have removed your custom settings?
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 22, 2016, 9:16am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/13 "2016-09-22T09:16:35Z")
Yes! ☹ Every 2 days the JVM heap goes over 90% and the nodes dies.
Some suggestions what to do? Do you need some stats, I can deliver it.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 22, 2016, 10:52am UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/14 "2016-09-22T10:52:20Z")
Can you please post your full current configuration? What does Marvel show with respect to heap usage over these 2 days before it reaches 90%?
---
### Author:  [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)
#### Post date: [September 25, 2016, 5:00pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/15 "2016-09-25T17:00:04Z")
I have suffered from the exact same problem. I have trying out various heap sizes, starting from 8 to 12 to 16 to 20 and none was sufficient, Each node has to be restarted every 2 days or so. Extremely long gc old , in the range of 2 minutes or more.
My latest experiment is to maximime heap to 30 GB and cluster is now on day 5. So this has definitely helped , but I am still seeing a small growth in heap usage. So sooner or later , I will probably have to restart the nodes.
I am also monitoring heap usage , if consistently over 90% , a script will restart the node automatically.
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 25, 2016, 6:33pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/16 "2016-09-25T18:33:39Z")
HI Kim, this is a very bad solution 🙂 I don't want to have this stress every day. This is impossible! It is some setting, but we haven't found it.
For example I have another project, where the index is only 2GB big, there are 30-40 reuquest per second and the server and heap ist working fine. Here I have one index only.
But on this problematic project I have 10 indexes , where everyone has 1GB of data. The heap is coming very fast over 90% and the nodes are not responding anymore.
I hope someone from elasticsearch team can help us to solve this problem!
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 25, 2016, 6:36pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/17 "2016-09-25T18:36:44Z")
Hi Christian, here is the data:
[[Album] Imgur: The magic of the Internet  ](https://imgur.com/a/OnBHk)
[[Album] Imgur: The magic of the Internet  ](https://imgur.com/a/xBtnV)
The settings of the node:
```
cluster.name: xxxx
node.name: xxxx-master
node.data: true
node.master: true
bootstrap.mlockall: true
index.merge.policy.merge_factor: 5
threadpool.index.queue_size: 10000
index.cache.query.enable: true
transport.tcp.compress: true;
index.store.type: mmapfs
network.bind_host: xxx.xxx.xxx.xxx
network.publish_host: xxx.xxx.xxx.xxx
network.host: xxx.xxx.xxx.xxx
discovery.zen.ping.unicast.hosts: ["xxx.xxx.xxx.xxx"]
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 10s
transport.tcp.port: 9300
http.port: 9200
http.max_content_length: 500mb
index.routing.allocation.disable_allocation: false
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
script.engine.groovy.inline.aggs: on
script.inline: on
script.indexed: on
index.max_result_window: 10000
```
As you suggest me, I have removed all the additional nodes and make only one node with 30GB of heap. This doesn't help! I become still after one day the 90% of the heap full and the node die. Not resonding anymore.
I have now 3x more heap space then my index is big.
Suggestion?
Thanks
Nik
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 25, 2016, 8:16pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/18 "2016-09-25T20:16:59Z")
You look to have too many shards, nearly 200 for only 6GB of data is going to be wasting a lot of resources.
What is in your slow log?
> [@renego](#):
>
> index.merge.policy.merge\_factor: 5
> threadpool.index.queue\_size: 10000
> index.store.type: mmapfs
> index.routing.allocation.disable\_allocation: false
You should remove all of those, they are either pointless or dangerous.
---
### Author:  [@renego](https://discuss.elastic.co/u/renego)
#### Post date: [September 25, 2016, 8:55pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/19 "2016-09-25T20:55:43Z")
Hi, where do you have seen this 200 ? I have actually 8 shards for each index.
Is the index.store.type: mmapfs not the best choise for an elasticsearch index?
Thanks
Nik
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 25, 2016, 9:45pm UTC](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232/20 "2016-09-25T21:45:27Z")
It's in the first picture you posted, in Marvel/Monitoring).
Have a read of [https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index-modules-store.html#file-system](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index-modules-store.html#file-system)
[Next page](https://discuss.elastic.co/t/please-suggest-performance-settings-for-my-elasticsearch-cluster/60232.md?page=2)