# Assistance with ELK **URL:** \ **Category:** Elasticsearch\ **Created:** [May 2, 2019, 7:38am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297 "2019-05-02T07:38:18Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 2, 2019, 7:38am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/1 "2019-05-02T07:38:18Z")
Hello I have been using ELK for quite some time now and currently facing issues with memory utilization and timeouts on Kibana when running long search queries. My elk is a single node cluster and running htop i see lot of ES processes with 57% of memory being consumed by ES. I understand ES uses many threadpools. The total memory assigned is 32GB. While we are planning for a long term solution for this, in the mean time can I get some suggestion on what can we do to reduce resource utilization. Thanks! ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 3, 2019, 7:53am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/2 "2019-05-03T07:53:24Z")
An i get errors as seen in the image whenever I try to get the data for larger time period like 1 year or so ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/50319032b66a396e2452604dfa7e3def1136de59.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 3, 2019, 8:07am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/3 "2019-05-03T08:07:09Z")
Sounds like your cluster is "overloaded" or can't keep up with all the shards you have? What is the output of: ```auto GET /_cat/indices?v GET /_cat/nodes?v GET /_cat/health?v GET /_cat/shards?v ``` ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 3, 2019, 8:39am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/4 "2019-05-03T08:39:30Z")
Thanks for the response David! We use a single node cluster as of now. Below is the requested detail. For the list of indices and shards i have just included the last line of the output to just include the count of them: ```auto GET /_cat/indices?v 755 yellow open logstash-2017.09.21 ECVj KguDR5OLYYbOA8cglw 5 1 217 0 270.2kb 270.2kb GET /_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 33 90 2 0.32 0.29 0.29 mdi * QMN7LjG GET /_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1556872627 08:37:07 elasticsearch yellow 1 1 3766 3766 0 0 3766 0 - 50.0% GET /_cat/shards?v 7532 logstash-2018.03.14 0 p STARTED 121 157.4kb 127.0.0.1 QMN7LjG 7533 logstash-2018.03.14 0 r UNASSIGNED ``` Thanks ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 3, 2019, 9:49am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/5 "2019-05-03T09:49:22Z")
So you have `3766` on a `32GB` HEAP. Correct? We recommend no more than 20 shards per gb of HEAP on a data node. At most 640 shards per node. Not including that your node is also managing the cluster (master node) and receiving the requests (coordinating node) coming from Kibana. For each request, you have to hit (and keep in memory) every single shard response to merge the result then and send back to Kibana the results. I think you can now imagine why you are seeing this behavior. How to fix? Multiple options (that you can combine): - Change the default number of shards to 1. You can have up to 50gb par shard. One of the index you have has `270.2kb` of data on 5 shards! - Use the shrink API - Use the rollup API to compress data with less granularity - Start new nodes (and if you don't need replicas, set the number of replicas to `0` first) - Remove old indices May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > If you are looking for practical guidelines around how many indices and shards to have in your cluster, this blog post will help you avoid common pitfalls. https://www.slideshare.net/slideshow/embed_code/key/vR0XKDq4TGa77z And [https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right) ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 4:56am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/6 "2019-05-06T04:56:26Z")
Hi David, Thanks much for the reply. I will try to use these options suggested and contact you back incase I have any further questions. Thanks! ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 5:11am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/7 "2019-05-06T05:11:25Z")
Just a quick question. To check number of shards this is the query: "GET \_cat/shards" From what I see with this, we have about 7533 shards. But I see the number 3766 unders 'shards' when I run "GET /\_cat/health?v" epoch timestamp cluster status node.total node.data shards pri relo init unassign pending\_tasks max\_task\_wait\_time active\_shards\_percent 1556872627 08:37:07 elasticsearch yellow 1 1 3766 3766 0 0 3766 0 - 50.0% And I guess you pointed out at the number 3766 from this query. Would like to know why the difference is and what is the actual no of shards that need to be considered. Thanks! ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 5:19am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/8 "2019-05-06T05:19:55Z")
Thats the count of replica shard too , i guess. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 6, 2019, 5:56am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/9 "2019-05-06T05:56:39Z")
Exact ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 7:33am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/10 "2019-05-06T07:33:11Z")
Hi David I have few queries regarding the same: 1. As I only have single node, enabling replica shard is of no use as even the replica set will remain in the same node. So I can set the number of replicas to 0. Please correct me if I'm wrong. a. As you recommend I can change the default no of shards to 1. And from the doc we have 2 options to change 'index.number\_of\_shards' and 'index.number\_of\_replicas' a. We can add below lines in elasticsearch.yml index.number\_of\_shards: 1 index.number\_of\_replicas: 0 OR b. We can make use of templates which will apply the changes to new indices. ## As of now i have one template which looks like this: ``` indent preformatted text by 4 spaces ``` ## { "logstash": { "order": 0, "version": 50001, "template": "logstash-_", "settings": { "index": { "refresh\_interval": "5s" } }, "mappings": { "default": { "\_all": { "enabled": true, "norms": false }, "dynamic\_templates": [{ "message\_field": { "path\_match": "message", "match\_mapping\_type": "string", "mapping": { "type": "text", "norms": false } } }, { "string\_fields": { "match": "_", "match\_mapping\_type": "string", "mapping": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword" } } } } }], "properties": { "@timestamp": { "type": "date", "include\_in\_all": false }, "@version": { "type": "keyword", "include\_in\_all": false }, "geoip": { "dynamic": true, "properties": { "ip": { "type": "ip" }, "location": { "type": "geo\_point" }, "latitude": { "type": "half\_float" }, "longitude": { "type": "half\_float" } } } } } }, "aliases": {} } } From the doc [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html) I can use below template to add "number\_of\_shards": 1 and "number\_of\_replicas": 0 PUT \_template/template\_1 { "template": "logstash-\*", "settings": { "number\_of\_shards": 1 "number\_of\_replicas": 0 }, "mappings": { "type1": { "\_source": { "enabled": false }, "properties": { "host\_name": { "type": "keyword" }, "created\_at": { "type": "date", "format": "EEE MMM dd HH:mm:ss Z YYYY" } } } } } Now my question is, If I use this new template, what happens to my existing template? Will there be any issues? OR should I make any changes to the second (new ) template with regards to the first and already existing template? And I understand that this change will apply only to new indices and if we want the changes to be applied to already created indices we have to make use of shrink API Thanks! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 6, 2019, 7:36am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/11 "2019-05-06T07:36:00Z")
Changing in `elasticsearch.yml` is not supported anymore. You need to do that in a template as you proposed. > And I understand that this change will apply only to new indices and if we want the changes to be applied to already created indices we have to make use of shrink API Correct. This will be applied to newly created indices. Existing indices won't change. ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 7:46am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/12 "2019-05-06T07:46:06Z")
Thanks David! Just to clarify, adding the new template (shared before) will not have any affect on the existing template correct? Regards! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 6, 2019, 8:02am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/13 "2019-05-06T08:02:28Z")
I'd overwrite the existing template. Or you need to create a new one which won't affect indeed the existing one. Templates will be "merged" in memory before being applied to index creation. You should give it a try in a test env to make sure that it works as you expect before pushing this to production. ---
**Author:** ![Eva](https://avatars.discourse-cdn.com/v4/letter/e/dbc845/32.png) [@Eva](https://discuss.elastic.co/u/Eva)\ **Post date:** [May 6, 2019, 8:06am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/14 "2019-05-06T08:06:35Z")
Alright, Thanks David. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 3, 2019, 8:06am UTC](https://discuss.elastic.co/t/assistance-with-elk/179297/15 "2019-06-03T08:06:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.