# ES at 100% CPU, millions of CLUSTER\_RECOVERED errors **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2016, 8:52am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982 "2016-10-01T08:52:20Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 8:52am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/1 "2016-10-01T08:52:20Z")
Hi, I've had ES running in my home lab for a while. It had been running happily for months, and I left it alone for a bit to work on some other things; when I came back to it I find that it's munching 100% CPU in the VM and not responding to anything other than basic GETs. I can't add/modify any data, and I can't stop the process from its systemd service once it's running, I have to kill -9. Here is an example of the errors which are happening (only in DEBUG mode, there's nothing significant coming up at the default logging level): > [2016-09-26 17:58:34,455][DEBUG][gateway] [myhost] [dom-ls-2016.09.14.21][0]: throttling allocation [[dom-ls-2016.09.14.21][0], node[null], [P], v[0], s[UNASSIGNED], unassigned\_info[[reason=CLUSTER\_RECOVERED], at[2016-09-26T16:58:17.887Z]]] to [[{myhost}{T\_VDx6VqR-i0DtlkhYxXKQ}{10.0.19.4}{10.0.19.4:9300}]] on primary allocation This is running on Debian Jessie, VM has access to 2 cores and 4GB RAM; just running a single primary node, nothing else in the cluster. Errors are occurring regardless of whether LS and Kibana are running. I don't know what to look for next. This COULD have been due to me snarling up the config somehow, but equally there are other things, e.g. my ESXi box doesn't have UPS and I've had more than one power cut. I've tried asking in the IRC channel on Freenode and unfortunately nobody there seemed to be able to help. Hopefully here I will find someone who knows how to help me fix it. Thanks in advance! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 9:16am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/2 "2016-10-01T09:16:33Z")
What version? ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 9:21am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/3 "2016-10-01T09:21:04Z")
My bad, 2.2.0. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 10:04am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/4 "2016-10-01T10:04:34Z")
What's in `_cat/pendingtasks`? ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 11:00am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/5 "2016-10-01T11:00:37Z")
32 of the following: > 1440 4.8s URGENT shard-started ([dom-ls-2016.05.21.08][0], node[EX\_53xOeRPKKu04ljCJAHw], [P], v[11], s[INITIALIZING], a[id=IGSzbbn-RLGwvOiTUWtpsw], unassigned\_info[[reason=CLUSTER\_RECOVERED], at[2016-10-01T11:11:30.741Z]]), reason [master {moose}{EX\_53xOeRPKKu04ljCJAHw}{10.0.19.4}{10.0.19.4:9300} marked shard as initializing, but shard state is [POST\_RECOVERY], mark shard as started] ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 11:05am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/6 "2016-10-01T11:05:37Z")
Anything interesting in the other `_cat` endpoints, eg recovery, allocation? ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 11:11am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/7 "2016-10-01T11:11:38Z")
I'm too new to ES to know what qualifies as interesting. All items in \_cat/recovery say 'store done' so I'm guessing nothing there. Output of \_cat/allocation is: > 2255 9.4gb 26.3gb 121.2gb 147.5gb 17 10.0.19.4 10.0.19.4 moose > 2409 UNASSIGNED ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 11:12am UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/8 "2016-10-01T11:12:49Z")
Ok problem is you need to reduce your shard count, quite a lot. You are going to be wasting _lots_ of resources just maintaining that many shards. ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 1:07pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/9 "2016-10-01T13:07:05Z")
My Logstash pipeline was set to create hourly indexes, with one shard per index. Warkolm has suggested that this is the cause of the problem and that the solution would be to re-index to monthly. However, although I have stopped ES from producing lots of Java heap errors (see [here](http://stackoverflow.com/questions/27359885/elasticsearch-outofmemoryerror-java-heap-space)), running the [reindex suggested](https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06) doesn't seem to be working - I should be getting dots output and I'm getting nothing. I suspect that this is an ES rather than logstash problem because when I have tried individual CURL requests, only the GETs have worked - PUT and DELETE fail. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 3:20pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/10 "2016-10-01T15:20:44Z")
What does your config look like? ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 1, 2016, 6:07pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/11 "2016-10-01T18:07:36Z")
Assuming you mean /etc/elasticsearch.yml: > cluster.name: elkmeup > node.name: moose > path.data: /opt/elasticsearch > path.logs: /opt/logs/elasticsearch > network.host: \_site\_ > network.bind\_host: ["_site_", "_local_"] That's it, nothing fancy. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2016, 10:25pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/12 "2016-10-01T22:25:13Z")
LS, that's where you think this issue is currently at? ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 2, 2016, 4:51pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/13 "2016-10-02T16:51:16Z")
No, there is definitely an issue with ES because I can't PUT/DELETE regardless of whether or not LS is running. ---
**Author:** ![scherma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scherma/32/23571_2.png) [@scherma](https://discuss.elastic.co/u/scherma)\ **Post date:** [October 4, 2016, 7:26pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/14 "2016-10-04T19:26:01Z")
Further troubleshooting: I have (eventually) become able to do POST/PUT/DELETE operations. I don't know for certain what made the difference but I suspect it was that ES simply needed enough time to process its recovery operations (I had been shutting it down between troubleshooting so as not to be stressing my VM host too much). I have now started reindexing to a more sensible frame (monthly instead of hourly), and I understand that shard count and index count have a significant impact on cluster performance and resource usage - but I would like some help understanding how and why this is the case, so that I can better judge how to manage index creation in future. Please could you enlighten me on this front? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/es-at-100-cpu-millions-of-cluster-recovered-errors/61982/15 "2017-07-05T22:15:00Z")