# Elasticsearch high cpu usage

**URL:** <https://discuss.elastic.co/t/elasticsearch-high-cpu-usage/18454>\
**Category:** Elasticsearch\
**Created:** [July 3, 2014, 2:32am UTC](https://discuss.elastic.co/t/elasticsearch-high-cpu-usage/18454 "2014-07-03T02:32:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vincent\_park](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincent_park/32/800_2.png) [@vincent\_park](https://discuss.elastic.co/u/vincent_park)\
**Post date:** [July 3, 2014, 2:32am UTC](https://discuss.elastic.co/t/elasticsearch-high-cpu-usage/18454/1 "2014-07-03T02:32:31Z")

</div>

Hi,  
I have 5 clustered nodes and each nodes have 1 replica.  
total document size is 216 M and 853,000 docs.  
I was suffering from very high CPU usage.  
every hours and every early morning about am 05:00 ~ am 09:00  
you can see my cacti graph.  
\<nabble\_img src="cpuhigh.jpg" border="0"/\>  
there is elasticsearch only on this server

I thought there are something wrong with es process.  
but there is a few server request at cpu peak time.  
and there is no cron job even.

every hours and every early morning about am 05:00 ~ am 09:00  
I don't know what's going on elasticsearch at this time!!  
somebody help me, tell me what happened in there. please..

$ ./elasticsearch -v  
Version: 1.1.1, Build: f1585f0/2014-04-16T14:27:12Z, JVM: 1.7.0\_55

$ java -version  
java version "1.7.0\_55"  
Java(TM) SE Runtime Environment (build 1.7.0\_55-b13)  
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

and I installed plugins on elasticsearch:  
HQ, bigdesk, head, kopf, sense

es log at cpu peak time:

[2014-07-03 08:01:00,045][DEBUG][action.search.type] [node1] [search][4], node[GJjzCrLvQQ-ZRRoqL13MrQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@451f9e7c] lastShard [true]  
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 300) on org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@68ab486b  
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)  
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)  
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)  
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:293)  
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:300)  
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:190)  
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)  
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)  
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:108)  
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)  
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)  
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)  
at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:98)  
at org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)  
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)  
at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)  
at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)  
at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)  
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)  
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)  
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)  
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)  
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)  
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)  
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)  
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)  
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)  
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)  
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)  
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)  
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)  
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  
at java.lang.Thread.run(Unknown Source)

---

<div class="post-metadata">

**Author:** ![Michael\_Hart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_hart/32/65937_2.png) [@Michael\_Hart](https://discuss.elastic.co/u/Michael_Hart)\
**Post date:** [July 3, 2014, 1:12pm UTC](https://discuss.elastic.co/t/elasticsearch-high-cpu-usage/18454/2 "2014-07-03T13:12:06Z")

</div>

A few things I can think of to look at. During this high CPU load, what's  
the:

- search rate
- index rate
- GC status (old and young) both number and duration
- IOPS

Are these nodes VM's? If so is there something else running on the other  
VM's?

That CPU load doesn't look too bad to me, it doesn't appear to be running  
flat out.

mike

On Thursday, July 3, 2014 7:37:57 AM UTC-4, vincent Park wrote:

> Hi,  
> I have 5 clustered nodes and each nodes have 1 replica.  
> total document size is 216 M and 853,000 docs.  
> I was suffering from very high CPU usage.  
> every hours and every early morning about am 05:00 ~ am 09:00  
> you can see my cacti graph.  
> [http://elasticsearch-users.115913.n3.nabble.com/file/n4059189/cpuhigh.jpg](http://elasticsearch-users.115913.n3.nabble.com/file/n4059189/cpuhigh.jpg)
> 
> there is elasticsearch only on this server
> 
> I thought there are something wrong with es process.  
> but there is a few server request at cpu peak time.  
> and there is no cron job even.
> 
> every hours and every early morning about am 05:00 ~ am 09:00  
> I don't know what's going on elasticsearch at this time!!  
> somebody help me, tell me what happened in there. please..
> 
> $ ./elasticsearch -v  
> Version: 1.1.1, Build: f1585f0/2014-04-16T14:27:12Z, JVM: 1.7.0\_55
> 
> $ java -version  
> java version "1.7.0\_55"  
> Java(TM) SE Runtime Environment (build 1.7.0\_55-b13)  
> Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
> 
> and I installed plugins on elasticsearch:  
> HQ, bigdesk, head, kopf, sense
> 
> es log at cpu peak time:
> 
> [2014-07-03 08:01:00,045][DEBUG][action.search.type] [node1]  
> [search][4], node[GJjzCrLvQQ-ZRRoqL13MrQ], [P], s[STARTED]: Failed to  
> execute [org.elasticsearch.action.search.SearchRequest@451f9e7c] lastShard  
> [true]  
> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:  
> rejected execution (queue capacity 300) on  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@68ab486b
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
> 
> ```
> at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) 
> at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) 
> at 
> 
> ```
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:293)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:300)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:190)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:108)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)  
> at  
> org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:98)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> 
> ```
> at 
> 
> ```
> 
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> 
> ```
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> at java.lang.Thread.run(Unknown Source) 
> 
> ```
> 
> --  
> View this message in context:  
> [http://elasticsearch-users.115913.n3.nabble.com/elasticsearch-high-cpu-usage-tp4059189.html](http://elasticsearch-users.115913.n3.nabble.com/elasticsearch-high-cpu-usage-tp4059189.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e0b55541-8f26-4ad3-80be-eb878a8f3844%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e0b55541-8f26-4ad3-80be-eb878a8f3844%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:18am UTC](https://discuss.elastic.co/t/elasticsearch-high-cpu-usage/18454/3 "2017-07-06T01:18:08Z")

</div>