# ES consume high cpu with threadlocal

**URL:** https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402
**Category:** Elasticsearch
**Created:** [January 29, 2018, 7:02am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402 "2018-01-29T07:02:19Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![rockiee281](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rockiee281](https://discuss.elastic.co/u/rockiee281)
#### Post date: [January 29, 2018, 7:02am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/1 "2018-01-29T07:02:19Z")

</div>

Hi all,  
I'm using es 2.4.2 with java8u60's G1GC. Recently we found the load often goes very high( 20~25 while our machine have 32 vocres). After profiling we find "ThreadLocalMap.expungeStaleEntry" is consuming almost 80%+ cpu. How this can be ? there is only 48 search thread, the search cannot cost so many time. Sorry I cannot upload my flame.svg, bellow is the snapshot.

 ![flame](https://us1.discourse-cdn.com/elastic/original/3X/5/3/53e504ef461e170b0e652ad0336d4ecac28acd5c.png)

---

<div class="post-metadata">

### Author: ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)
#### Post date: [January 29, 2018, 8:14am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/2 "2018-01-29T08:14:10Z")

</div>

whoaa.. flame graphs!

if you stop the search , did you get better cpu idle time?

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [January 29, 2018, 8:19am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/3 "2018-01-29T08:19:15Z")

</div>

You are using an old Elasticsearch version (two major versions behind), you are using an old java version and a not recommended GC. Everyone of those components has seen an **huge** amount of fixes in the last years, so it is super tedious to try to debug this.

For example newer Elasticsearch versions have that thread local removed since almost 1.5 years, see [https://github.com/elastic/elasticsearch/pull/20778](https://github.com/elastic/elasticsearch/pull/20778)

---

<div class="post-metadata">

### Author: ![rockiee281](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rockiee281](https://discuss.elastic.co/u/rockiee281)
#### Post date: [January 29, 2018, 11:38am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/4 "2018-01-29T11:38:57Z")

</div>

Thanks for your reply! So I think the best choice is to upgrade ES to 5.x ?  
I 've update the jdk version to jdk8u161 on one of my cluster and wander what will happen next

---

<div class="post-metadata">

### Author: ![rockiee281](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rockiee281](https://discuss.elastic.co/u/rockiee281)
#### Post date: [January 29, 2018, 11:46am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/5 "2018-01-29T11:46:33Z")

</div>

BTW, which version will you suggest for product enviroment?

---

<div class="post-metadata">

### Author: ![rockiee281](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rockiee281](https://discuss.elastic.co/u/rockiee281)
#### Post date: [January 29, 2018, 11:54am UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/6 "2018-01-29T11:54:42Z")

</div>

As the load is too high, I restart my es node one by one

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [January 29, 2018, 12:17pm UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/7 "2018-01-29T12:17:36Z")

</div>

The latest **6**.x release is what I recommend. Also you cannot just go from 2.x to 6.x - this requires planning, as the data format is not compatible as well as queries and mapping configuration. So make sure test everything in preproduction systems before doing this on your live system.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 26, 2018, 12:17pm UTC](https://discuss.elastic.co/t/es-consume-high-cpu-with-threadlocal/117402/8 "2018-02-26T12:17:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
