# 0.90.6 upgrade - ES very slow/high CPU usage when recovering some shard **URL:** \ **Category:** Elasticsearch\ **Created:** [November 5, 2013, 3:29pm UTC](https://discuss.elastic.co/t/0-90-6-upgrade-es-very-slow-high-cpu-usage-when-recovering-some-shard/14267 "2013-11-05T15:29:48Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [November 5, 2013, 3:29pm UTC](https://discuss.elastic.co/t/0-90-6-upgrade-es-very-slow-high-cpu-usage-when-recovering-some-shard/14267/1 "2013-11-05T15:29:48Z")
We have a 3-level (grandparent, parent, child) index with 100 GB index size. After upgrading to ES 0.90.6, we noticed that ES became really slow with a persistent high CPU usage even after recovery completed. It seems that some post-processing after shard recovery was really heavy and taking high toll on ES. We had to eventually shut down ES and remove this index from local directory. I generated couple hot threads dump(9 minutes apart), and one common theme is "org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:149)". I posted complete gist at [https://gist.github.com/ajhalani/7320575](https://gist.github.com/ajhalani/7320575) Would much appreciate any help to understand the issue here. thanks!! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Nick\_Zadrozny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nick_zadrozny/32/1150_2.png) [@Nick\_Zadrozny](https://discuss.elastic.co/u/Nick_Zadrozny)\ **Post date:** [November 5, 2013, 4:50pm UTC](https://discuss.elastic.co/t/0-90-6-upgrade-es-very-slow-high-cpu-usage-when-recovering-some-shard/14267/2 "2013-11-05T16:50:10Z")
All of those FieldCache calls, if you look a little further down the trace, are the result of search calls against the index. I’m guessing your searches rely a lot on the field cache (looks like facets), which were cleared out when Elasticsearch restarted. The simplest explanation is that you’re simply seeing the efforts of Elasticsearch to warm up its caches, although that would more likely manifest as system load than CPU per se. Are you serving a large volume of traffic right away? Do you perhaps have a lot of warming queries? How much memory do your servers have, and how much have you allocated to Elasticsearch specifically? On Tue, Nov 5, 2013 at 8:29 AM, Ankush Jhalani [ankush.jhalani@gmail.com](mailto:ankush.jhalani@gmail.com)wrote: > We have a 3-level (grandparent, parent, child) index with 100 GB index > size. After upgrading to ES 0.90.6, we noticed that ES became really slow > with a persistent high CPU usage even after recovery completed. It seems > that some post-processing after shard recovery was really heavy and taking > high toll on ES. We had to eventually shut down ES and remove this index > from local directory. > > I generated couple hot threads dump(9 minutes apart), and one common theme > is "org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:149)". > I posted complete gist at [ES Hot threads dump · GitHub](https://gist.github.com/ajhalani/7320575) > > Would much appreciate any help to understand the issue here. thanks!! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Nick Zadrozny Cofounder, CEO One More Cloud [websolr.com](http://websolr.com) • [bonsai.io](http://bonsai.io) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [November 5, 2013, 8:51pm UTC](https://discuss.elastic.co/t/0-90-6-upgrade-es-very-slow-high-cpu-usage-when-recovering-some-shard/14267/3 "2013-11-05T20:51:51Z")
Nick you seem to be right. We had a huge registered warmer(known to take a lot of time) for this index, and looks like there's a bug in 0.90.6 { [Remove Index Reader warmer introduced in 0.90.6 · Issue #4078 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/4078)}. We were not serving any traffic. thanks!! On Tuesday, November 5, 2013 10:29:48 AM UTC-5, Ankush Jhalani wrote: > We have a 3-level (grandparent, parent, child) index with 100 GB index > size. After upgrading to ES 0.90.6, we noticed that ES became really slow > with a persistent high CPU usage even after recovery completed. It seems > that some post-processing after shard recovery was really heavy and taking > high toll on ES. We had to eventually shut down ES and remove this index > from local directory. > > I generated couple hot threads dump(9 minutes apart), and one common theme > is "org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:149)". > I posted complete gist at [ES Hot threads dump · GitHub](https://gist.github.com/ajhalani/7320575) > > Would much appreciate any help to understand the issue here. thanks!! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/0-90-6-upgrade-es-very-slow-high-cpu-usage-when-recovering-some-shard/14267/4 "2017-07-06T02:08:41Z")
Closed on July 6, 2017, 2:08am UTC