# Problems in my Cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [May 17, 2017, 7:23am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057 "2017-05-17T07:23:11Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 7:23am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/1 "2017-05-17T07:23:11Z")
Hi, We are experiencing some troubles with our cluster. When we come into the office on monday, one or two of our nodes are gone including the master. I also get this message in the logs: `org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s` From what I read here in the forum that could be because I have to many shards, which is highly possible when I look at my clusterhealth. > { > "cluster\_name" : "elasticsearch", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 5, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 9100, > "active\_shards" : 23225, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 100.0 > } We have in all indices approximately 20 million hits. I really appreciate any approach on improving the stability of my cluster, because getting my clusterhealth back to green is a pain in the neck kind regards Andy ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 17, 2017, 7:30am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/2 "2017-05-17T07:30:49Z")
You have far too many shards for a cluster that size. You need to revise you sharing strategy and bring that down by at least an order of magnitude or so. Aim to have an average shard size between a few GB and a few tens of GB. ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 7:34am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/3 "2017-05-17T07:34:43Z")
Where can I check the size of a shard? And which configuration would you recommend for the case I have, if I am allowed to ask? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 17, 2017, 7:37am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/4 "2017-05-17T07:37:29Z")
you can check shard and index size through the `_cat/indices` and `_cat/shards` APIs. What type of data do you have in the cluster? What is your current sharding strategy? If you are using time-based indices, what is your retention period? Which version of Elasticsearch are you using? ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 7:49am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/5 "2017-05-17T07:49:35Z")
Thank you. Ok my biggest index is something around 18Gb... and some of my shards are around 1,5Gb. We are using it for Apache logfiles, some windows service logs and since a month or so the output of our docker containers. We create a new index for everyday, but we have 9 indices. We are running 5.0.1. I am not quite sure what you meant with sharing strategy, but if it is the shard and replica config, there it is: > ``` > { > "logstash-2017.05.16" : { > "settings" : { > "index" : { > "refresh_interval" : "5s", > "number_of_shards" : "5", > "provided_name" : "logstash-2017.05.16", > "creation_date" : "1494892819101", > "number_of_replicas" : "1", > "uuid" : "pPFz1d3EQEe6XY-dlw344w", > "version" : { > "created" : "5000199" > } > } > } > } > } > > ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 17, 2017, 7:56am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/6 "2017-05-17T07:56:16Z")
That was supposed to be sharding, not sharing. The biggest index seems OK, but probably do not need 5 primary shards. Adjust the number of primary shards and do not use the default of 5 for very small indices. Also consider consolidating small indices and/or using weekly or even monthly indices instead of daily. ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 7:59am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/7 "2017-05-17T07:59:34Z")
If I am not completely wrong I can't change the shard size to anything smaller without removing the index? But first of all thank you for your help. You already helped me a lot. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 17, 2017, 8:03am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/8 "2017-05-17T08:03:13Z")
As you are on Elasticsearch 5.x, the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/indices-shrink-index.html) can help you get from 5 to 1 shard per index. You may also be able to reduce the number of replicas you have configured in order to bring the shard count down. Beyond that, and I think you will need to reduce the shard count further than that, you will need to reindex data. This can take time, but do change the settings for newly created indices so that you generate fewer new shards per day right away. ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 8:11am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/9 "2017-05-17T08:11:36Z")
Can you give me advice on reindexing as well? I have never done that before. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 17, 2017, 8:15am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/10 "2017-05-17T08:15:44Z")
You should be able to use the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-reindex.html#docs-reindex) to do this. ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 17, 2017, 8:20am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/11 "2017-05-17T08:20:34Z")
[![](https://media.tenor.co/images/d1bcd7be62c1f27d4e6978ad3cd2ab29/tenor.gif) ](https://media.tenor.co/images/d1bcd7be62c1f27d4e6978ad3cd2ab29/tenor.gif) ---
**Author:** ![McElroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcelroy/32/24639_2.png) [@McElroy](https://discuss.elastic.co/u/McElroy)\ **Post date:** [May 23, 2017, 6:03am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/12 "2017-05-23T06:03:47Z")
I think this looks a whole lot better. Thank you again for your help > { > "cluster\_name" : "elasticsearch", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 5, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 2866, > "active\_shards" : 5977, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 100.0 > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2017, 6:04am UTC](https://discuss.elastic.co/t/problems-in-my-cluster/86057/13 "2017-06-20T06:04:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.