# Elasticsearch very high load - 100% cpu

**URL:** <https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331>\
**Category:** Elasticsearch\
**Created:** [August 5, 2016, 10:11am UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331 "2016-08-05T10:11:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![DDE](https://avatars.discourse-cdn.com/v4/letter/d/bb73d2/32.png) [@DDE](https://discuss.elastic.co/u/DDE)\
**Post date:** [August 5, 2016, 10:11am UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331/1 "2016-08-05T10:11:52Z")

</div>

Hi,

We have our production cluster at 100% with load average of 100.

- JDK 1.7u55\_64bits
- ELS 1.7.5
- 9 nodes 30GB heap
- centos 6.5-6.6

in ELS : 1700 indices / 3200 shards / 5B docs / 5 TB  
For the requests : Only Kibana 3 with ~50 dashboards (so no aggragetion, only facets)  
We have 5-10 request by second  
10 000 indexations/second with average document size \<1k  
The global cpu load is 5% in busy time

In fact the search queue is exhausted and the active search pool is full.  
The curent requests never finish.  
The indexing still works fine.

We must restart the cluster to resolve the problem.  
The problem is now random, everything worked fine for months and now it crashed everyday.

- Can we put a global request timeout ?
- Anybody already had this problem ?
- what can we do apart restarting the cluster ?

Here an extract of the hot\_threads

54.2% (270.7ms out of 500ms) cpu usage by thread 'elasticsearch[server1][search][T#14]'  
10/10 snapshots sharing following 11 elements  
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:51)  
org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:350)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:66)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:338)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:324)  
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)  
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)

50.9% (254.2ms out of 500ms) cpu usage by thread 'elasticsearch[server1][search][T#6]'  
10/10 snapshots sharing following 11 elements  
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:51)  
org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:350)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:66)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:338)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:324)  
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)  
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
java.lang.Thread.run(Thread.java:745)

50.6% (252.8ms out of 500ms) cpu usage by thread 'elasticsearch[server1][search][T#21]'  
10/10 snapshots sharing following 13 elements  
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.\_skipWSOrEnd(UTF8StreamJsonParser.java:2728)  
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:652)  
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:51)  
org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:350)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187)  
org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:66)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:338)  
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:324)  
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)

thx for any help

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 5, 2016, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331/2 "2016-08-05T10:31:30Z")

</div>

What type of data do you have in the cluster? How much data do you have in the cluster? How many indices/shards? What is your indexing and query rates? What type of queries do you run?

---

<div class="post-metadata">

**Author:** ![DDE](https://avatars.discourse-cdn.com/v4/letter/d/bb73d2/32.png) [@DDE](https://discuss.elastic.co/u/DDE)\
**Post date:** [August 5, 2016, 11:08am UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331/3 "2016-08-05T11:08:35Z")

</div>

in ELS : 1700 indices / 3200 shards / 5B docs / 5 TB  
For the requests : Only Kibana 3 with ~50 dashboards (so no aggragetion, only facets)  
We have 5-10 request by second  
10 000 indexations/second with average document size \<1k  
The global cpu load is 5% in busy time

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 6, 2016, 5:26am UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331/4 "2016-08-06T05:26:01Z")

</div>

I'd suggest you have too many shards, which will be adding to heap pressure.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:29pm UTC](https://discuss.elastic.co/t/elasticsearch-very-high-load-100-cpu/57331/5 "2017-07-05T22:29:40Z")

</div>