# CPU usages 90% and ES hotthreads dump

**URL:** <https://discuss.elastic.co/t/cpu-usages-90-and-es-hotthreads-dump/14005>\
**Category:** Elasticsearch\
**Created:** [October 18, 2013, 3:45am UTC](https://discuss.elastic.co/t/cpu-usages-90-and-es-hotthreads-dump/14005 "2013-10-18T03:45:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sujan\_dutta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujan_dutta/32/2041_2.png) [@sujan\_dutta](https://discuss.elastic.co/u/sujan_dutta)\
**Post date:** [October 18, 2013, 3:45am UTC](https://discuss.elastic.co/t/cpu-usages-90-and-es-hotthreads-dump/14005/1 "2013-10-18T03:45:13Z")

</div>

Hi everybody

i have a ES cluster of 2 node (8 GB) machine.  
each ES node assigned 4 GB memory. I have 1 index with 10 shard and 1  
replica. this index has around 35 millions documents of 20 GB.  
i have 2 more indexes, which have few hundreds of documents.

Suddenly in 1 node cpu usages is 90% and indexing/search operations  
response time increased to 4-5 secs.

i have taken hotthread infomation, if anybody can analyse it and inform me  
that i need to increase memory of the node or add other node horizontally.  
Or any other things i need to rectify this problem.

thanks - sujan

## hot threads:

:::  
[p\_gossamer\_v2\_data\_master\_client][5AkaMXO7ReWLiLHaOKqu7g][inet[/10.236.184.30:9300]]{transact=tag\_transact\_v01,  
sandbox=tag\_sandbox\_v01, storetype=primary, max\_local\_storage\_nodes=1,  
master=true, metadata=tag\_metadata\_v01}

69.8% (348.7ms out of 500ms) cpu usage by thread  
'elasticsearch[p\_gossamer\_v2\_data\_master\_client][search][T#524]'  
3/10 snapshots sharing following 10 elements

org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)

org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)  
7/10 snapshots sharing following 9 elements  
sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)  
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)

60.6% (302.7ms out of 500ms) cpu usage by thread  
'elasticsearch[p\_gossamer\_v2\_data\_master\_client][search][T#520]'  
3/10 snapshots sharing following 10 elements

org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)

org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)

org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)  
6/10 snapshots sharing following 9 elements  
sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)  
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)  
unique snapshot

org.elasticsearch.search.internal.InternalSearchResponse.(InternalSearchResponse.java:51)

org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:354)

org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:86)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:229)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:208)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:205)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:244)

org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)

org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)

```
5.5% (27.4ms out of 500ms) cpu usage by thread

```

'elasticsearch[p\_gossamer\_v2\_data\_master\_client][index][T#3451]'  
2/10 snapshots sharing following 3 elements

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)  
8/10 snapshots sharing following 9 elements  
sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)  
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:722)

:::  
[p\_gossamer\_v2\_data\_master\_r1][\_hotrVvST\_CAwJw6f3wd-A][inet[/10.143.162.238:9300]]{transact=tag\_transact\_v01,  
sandbox=tag\_sandbox\_v01, storetype=replica, max\_local\_storage\_nodes=1,  
master=true, metadata=tag\_metadata\_v01}

```
3.6% (17.9ms out of 500ms) cpu usage by thread

```

'elasticsearch[p\_gossamer\_v2\_data\_master\_r1][index][T#4580]'  
10/10 snapshots sharing following 9 elements  
sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)  
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 20, 2013, 3:37pm UTC](https://discuss.elastic.co/t/cpu-usages-90-and-es-hotthreads-dump/14005/2 "2013-10-20T15:37:54Z")

</div>

Hey,

what kind of operation are you executing when this happens? Are you trying  
to return a huge amount of documents in your search requests? Anything else  
which might help as well?

--Alex

On Fri, Oct 18, 2013 at 5:45 AM, Sujan Dutta [sujandutta@gmail.com](mailto:sujandutta@gmail.com) wrote:

> Hi everybody
> 
> i have a ES cluster of 2 node (8 GB) machine.  
> each ES node assigned 4 GB memory. I have 1 index with 10 shard and 1  
> replica. this index has around 35 millions documents of 20 GB.  
> i have 2 more indexes, which have few hundreds of documents.
> 
> Suddenly in 1 node cpu usages is 90% and indexing/search operations  
> response time increased to 4-5 secs.
> 
> i have taken hotthread infomation, if anybody can analyse it and inform me  
> that i need to increase memory of the node or add other node horizontally.  
> Or any other things i need to rectify this problem.
> 
> thanks - sujan
> 
> ## hot threads:
> 
> :::  
> [p\_gossamer\_v2\_data\_master\_client][5AkaMXO7ReWLiLHaOKqu7g][inet[/10.236.184.30:9300]]{transact=tag\_transact\_v01,  
> sandbox=tag\_sandbox\_v01, storetype=primary, max\_local\_storage\_nodes=1,  
> master=true, metadata=tag\_metadata\_v01}
> 
> 69.8% (348.7ms out of 500ms) cpu usage by thread  
> 'elasticsearch[p\_gossamer\_v2\_data\_master\_client][search][T#524]'  
> 3/10 snapshots sharing following 10 elements
> 
> org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)  
> 7/10 snapshots sharing following 9 elements  
> sun.misc.Unsafe.park(Native Method)
> 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)
> 
> 60.6% (302.7ms out of 500ms) cpu usage by thread  
> 'elasticsearch[p\_gossamer\_v2\_data\_master\_client][search][T#520]'  
> 3/10 snapshots sharing following 10 elements
> 
> org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
> 
> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)  
> 6/10 snapshots sharing following 9 elements  
> sun.misc.Unsafe.park(Native Method)
> 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)  
> unique snapshot
> 
> org.elasticsearch.search.internal.InternalSearchResponse.(InternalSearchResponse.java:51)
> 
> org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:354)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryAndFetchAction.java:86)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:229)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:208)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:205)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:244)
> 
> org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)
> 
> ```
> 5.5% (27.4ms out of 500ms) cpu usage by thread
> 
> ```
> 
> 'elasticsearch[p\_gossamer\_v2\_data\_master\_client][index][T#3451]'  
> 2/10 snapshots sharing following 3 elements
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)  
> 8/10 snapshots sharing following 9 elements  
> sun.misc.Unsafe.park(Native Method)
> 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:722)
> 
> :::  
> [p\_gossamer\_v2\_data\_master\_r1][\_hotrVvST\_CAwJw6f3wd-A][inet[/10.143.162.238:9300]]{transact=tag\_transact\_v01,  
> sandbox=tag\_sandbox\_v01, storetype=replica, max\_local\_storage\_nodes=1,  
> master=true, metadata=tag\_metadata\_v01}
> 
> ```
> 3.6% (17.9ms out of 500ms) cpu usage by thread
> 
> ```
> 
> 'elasticsearch[p\_gossamer\_v2\_data\_master\_r1][index][T#4580]'  
> 10/10 snapshots sharing following 9 elements  
> sun.misc.Unsafe.park(Native Method)
> 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:11am UTC](https://discuss.elastic.co/t/cpu-usages-90-and-es-hotthreads-dump/14005/3 "2017-07-06T02:11:35Z")

</div>