# Recovering a poor performing Elasticsearch install **URL:** \ **Category:** Elasticsearch\ **Created:** [September 28, 2017, 4:02pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146 "2017-09-28T16:02:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![paulglavin](https://avatars.discourse-cdn.com/v4/letter/p/b38774/32.png) [@paulglavin](https://discuss.elastic.co/u/paulglavin)\ **Post date:** [September 28, 2017, 4:02pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146/1 "2017-09-28T16:02:57Z")
I've been trying to figure out why an elasticsearch install I've inherited is running poorly. My original thoughts were both hardware and cluster sizing. I'm in the position of not knowing alot of things about the install, I don't know how much data is being sent to the cluster, what is retained and what our query types are. I've got x-pack installed now and working but that has highlighted a major performance issue to the point that Kibana is constantly showing the status page with Status of Red, I can't get to any of the other apps as the status page is always loaded! ![status](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7b29d628ccefd3a9ca57764015c8a699259c0b80.JPG) I started to dig further into the timeouts and my cluster health showed: { "cluster\_name" : "elasticsearch", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 2224, "active\_shards" : 2224, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 2223, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 50.0112435349674 } The size of unassigned shards was a worry curl -u elastic -XGET 172.20.3.247:9200/\_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED utm-2017.08.11 4 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.11 1 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.11 2 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.11 3 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.11 0 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.06.16 4 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.06.16 1 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.06.16 2 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.06.16 3 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.06.16 0 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.09.08 4 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.09.08 1 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.09.08 2 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.09.08 3 r UNASSIGNED CLUSTER\_RECOVERED winlogbeat-2017.09.08 0 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.14 4 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.14 1 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.14 2 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.14 3 r UNASSIGNED CLUSTER\_RECOVERED utm-2017.08.14 0 r UNASSIGNED CLUSTER\_RECOVERED I'm stuck as to where to even start to size this cluster and I don't know where to go next to get the install usable. The node is running in AWS, it's on a m4.xlarge I can throw more resource at it short term but I want to get it into a better state long term. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 28, 2017, 4:32pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146/2 "2017-09-28T16:32:41Z")
An m4.xlarge has 16GB of RAM, which means that you probably have a 8GB heap, assuming best practices has been followed. Given the number of shards you have on this node (way too many for a node that size) I suspect you may be suffering from heap pressure. This is a quite common problem (there are quite a few posts here around this just in the last few days), and the reason I wrote a [blog post about shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). As outlined in this blog post, I would recommend that you reduce the number of shards dramatically. If you are on Elasticsearch 5.x you may be able to use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-shrink-index.html), but you may also need to reindex data. If your cluster is in a bad state and does not allow this, you may need to close or delete some indices before you can do this. You can also scale up/out the cluster to get more resources. ---
**Author:** ![paulglavin](https://avatars.discourse-cdn.com/v4/letter/p/b38774/32.png) [@paulglavin](https://discuss.elastic.co/u/paulglavin)\ **Post date:** [September 28, 2017, 6:47pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146/3 "2017-09-28T18:47:37Z")
It was upgraded from a t2.medium last night to try and make it usable so settings may not have been changed to reflect that upgrade. It's running 5.5 at the moment. Looking at the shards, if I'm reading the output of the command I used, it looks like I have 5 shards per index. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 28, 2017, 6:55pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146/4 "2017-09-28T18:55:42Z")
Have a look at the configuration and ensure that you have heap size set to 8GB (assuming nothing else is running on that host). That may give you some additional headroom so you can address the sharing issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2017, 6:55pm UTC](https://discuss.elastic.co/t/recovering-a-poor-performing-elasticsearch-install/102146/5 "2017-10-26T18:55:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.