# CPU at 99% - Need help understanding hot\_threads ! pls
**URL:** https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171
**Category:** Elasticsearch
**Created:** [February 18, 2016, 5:42pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171 "2016-02-18T17:42:46Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Camilo\_Sierra](https://discuss.elastic.co/u/Camilo_Sierra)
#### Post date: [February 18, 2016, 5:42pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/1 "2016-02-18T17:42:46Z")
85.2% (426.1ms out of 500ms) cpu usage by thread 'elasticsearch[Data 1-hot][bulk][T#1]'
3/10 snapshots sharing following 19 elements
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:685)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604)
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:489)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:409)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:515)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:232)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 18 elements
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:685)
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604)
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:489)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
---
### Author:  [@Camilo\_Sierra](https://discuss.elastic.co/u/Camilo_Sierra)
#### Post date: [February 18, 2016, 5:42pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/2 "2016-02-18T17:42:52Z")
```
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:409)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:515)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:232)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 11 elements
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:409)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:515)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:232)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
```
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [February 18, 2016, 5:47pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/3 "2016-02-18T17:47:15Z")
how many shards in your cluster?
are you executing snapshot in this time?
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [February 18, 2016, 5:47pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/4 "2016-02-18T17:47:30Z")
and your free disk?
---
### Author:  [@Camilo\_Sierra](https://discuss.elastic.co/u/Camilo_Sierra)
#### Post date: [February 18, 2016, 5:55pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/5 "2016-02-18T17:55:20Z")
we have 433 shards in the cluster,
but only 3 shards in the index that makes trouble, this 3 shards are splited in 3 nodes. this is the only index that indexes documents
and dont execute any snapshot in this cluster.
and in disk we have 26% used !
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [February 18, 2016, 7:23pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/6 "2016-02-18T19:23:47Z")
is shards in same index?
there's replicas for this indexes?
are you change bulk queue and size?
---
### Author:  [@Camilo\_Sierra](https://discuss.elastic.co/u/Camilo_Sierra)
#### Post date: [February 19, 2016, 2:09pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/7 "2016-02-19T14:09:18Z")
Yes same index, at the beginig they have replicad but i deleted when the problem started.
but i dont change the size or queue, but for the 18 first days it works and yesterday the traffic was normal but the CPU skyrocket
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [February 19, 2016, 4:06pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/8 "2016-02-19T16:06:14Z")
Maybe the Elasticsearch host not support the size of your cluster.
How much RAM and CPU you are using?
In my case, I had a cluster with 10 hosts and 7Gbs of RAM each. I had a index with more than 100Gbs of data and my cluster down everyday. I decided use another kind of host in aws, with more RAM and CPU and less indexes in same time in the cluster.
Have you considered put more CPU and RAM?
Are you verified the bulk.queue and bulk.size?
/\_cat/thread\_pool?v&h=name,host,bulk.active,bulk.rejected,bulk.completed,bulk.queue,bulk.queueSize
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:14pm UTC](https://discuss.elastic.co/t/cpu-at-99-need-help-understanding-hot-threads-pls/42171/9 "2017-07-05T23:14:49Z")