# Logspam caused by getting index status in 1.3.3 **URL:** \ **Category:** Elasticsearch\ **Created:** [September 29, 2014, 9:43pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006 "2014-09-29T21:43:00Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![schmichael\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/schmichael_2/32/1249_2.png) [@schmichael\_2](https://discuss.elastic.co/u/schmichael_2)\ **Post date:** [September 29, 2014, 9:43pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/1 "2014-09-29T21:43:00Z")
It appears fetching index status information in Elasticsearch 1.3.3 causes logspam and massive CPU util. All of the loglines are of the form: 2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status [local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC\_wwXcjg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4] org.elasticsearch.transport.RemoteTransportException: [remote-alias][inet[/192.168.115.43:9300]][indices/status/s] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 100) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105 at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) This behavior did not occur in 1.1.1. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [September 29, 2014, 9:50pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/2 "2014-09-29T21:50:52Z")
Mind running hot\_threads and looking for something telling? That exception is more describing the symptom then the problem. On Mon, Sep 29, 2014 at 5:43 PM, schmichael [michael.schurter@gmail.com](mailto:michael.schurter@gmail.com) wrote: > It appears fetching index status information in Elasticsearch 1.3.3 causes > logspam and massive CPU util. > > All of the loglines are of the form: > > 2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status > [local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC\_wwXcjg], [R], > s[STARTED]: failed to executed > [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4 > ] > org.elasticsearch.transport.RemoteTransportException: > [remote-alias][inet[/192.168.115.43:9300]][indices/status/s] > Caused by: > org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: > rejected execution (queue capacity 100) on > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105 > at > org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > This behavior did not occur in 1.1.1. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3xhnfOrzCRGa7B3dxotE\_%2BeerU8gij9RAk05tw\_\_aajg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3xhnfOrzCRGa7B3dxotE_%2BeerU8gij9RAk05tw__aajg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![schmichael\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/schmichael_2/32/1249_2.png) [@schmichael\_2](https://discuss.elastic.co/u/schmichael_2)\ **Post date:** [September 29, 2014, 9:52pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/3 "2014-09-29T21:52:38Z")
The behavior does not seem to occur with 1.3.2. Both kopf and bigdesk cause the error in 1.3.3, and I was running the versions of them appropriate for 1.3. On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote: > It appears fetching index status information in Elasticsearch 1.3.3 causes > logspam and massive CPU util. > > All of the loglines are of the form: > > 2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status > [local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC\_wwXcjg], [R], > s[STARTED]: failed to executed > [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4] > org.elasticsearch.transport.RemoteTransportException: > [remote-alias][inet[/192.168.115.43:9300]][indices/status/s] > Caused by: > org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: > rejected execution (queue capacity 100) on > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105 > at > org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > This behavior did not occur in 1.1.1. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 29, 2014, 10:18pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/4 "2014-09-29T22:18:02Z")
There is already an issue opened for this: > > > Hi. > > While /\_status is deprecated and should be replaced with /\_recovery, there …seems to be a regression between 1.3.2 and 1.3.3 when a cluster has lots of shards. > > The /\_status-request fails with e.g: > > \`\`\` > "failures" : \[ { > "index" : "foo-0104", > "shard" : 0, > "reason" : "BroadcastShardOperationFailedException\[\[foo-0104\]\[0\] \]; nested: EsRejectedExecutionException\[rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c\]; " > } \] > } > \`\`\` > > Elasticsearch logs > > \`\`\` > \[2014-09-29 19:05:09,657\]\[DEBUG\]\[action.admin.indices.status\] \[Sputnik\] \[foo-0104\]\[0\], node\[LWIGdaF4QdWwLEz2peydqQ\], \[P\], s\[STARTED\]: failed to executed \[org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@746ef917\] > org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c > at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) > at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:166) > at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.start(TransportBroadcastOperationAction.java:150) > at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:70) > at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46) > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:65) > at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:71) > at org.elasticsearch.client.support.AbstractIndicesAdminClient.status(AbstractIndicesAdminClient.java:429) > at org.elasticsearch.rest.action.admin.indices.status.RestIndicesStatusAction.handleRequest(RestIndicesStatusAction.java:62) > at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:66) > at org.elasticsearch.rest.RestController.executeHandler(RestController.java:177) > at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:160) > at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121) > at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83) > at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:301) > at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44) > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) > at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > \`\`\` > > This reliably reproduces the issue with a clean install of 1.3.3: > \- Create 200 indexes, e.g. \`\>\>\> for i in range(0, 200): requests.put('http://localhost:9200/foo-%04i' % i, data='{"index": {"number\_of\_shards": 1, "number\_of\_replicas": 0}}')\` > \- \`curl localhost:9200/\_status\` > > This, in turn, causes dashboards like Kopf and Head to break. Jörg On Mon, Sep 29, 2014 at 11:52 PM, schmichael [michael.schurter@gmail.com](mailto:michael.schurter@gmail.com) wrote: > The behavior does not seem to occur with 1.3.2. > > Both kopf and bigdesk cause the error in 1.3.3, and I was running the > versions of them appropriate for 1.3. > > On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote: > > > It appears fetching index status information in Elasticsearch 1.3.3 > > causes logspam and massive CPU util. > > > > All of the loglines are of the form: > > > > 2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status > > [local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC\_wwXcjg], [R], > > s[STARTED]: failed to executed [org.elasticsearch.action. > > admin.indices.status.IndicesStatusRequest@7cc1c6f4] > > org.elasticsearch.transport.RemoteTransportException: > > [remote-alias][inet[/192.168.115.43:9300]][indices/status/s] > > Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: > > rejected execution (queue capacity 100) on org.elasticsearch.transport. > > netty.MessageChannelHandler$RequestHandler@4cd03105 > > at org.elasticsearch.common.util.concurrent.EsAbortPolicy. > > rejectedExecution(EsAbortPolicy.java:62) > > at java.util.concurrent.ThreadPoolExecutor.reject( > > ThreadPoolExecutor.java:821) > > at java.util.concurrent.ThreadPoolExecutor.execute( > > ThreadPoolExecutor.java:1372) > > at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest( > > MessageChannelHandler.java:219) > > at org.elasticsearch.transport.netty.MessageChannelHandler. > > messageReceived(MessageChannelHandler.java:111) > > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. > > handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > sendUpstream(DefaultChannelPipeline.java:564) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ > > DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > > 791) > > at org.elasticsearch.common.netty.channel.Channels. > > fireMessageReceived(Channels.java:296) > > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. > > unfoldAndFireMessageReceived(FrameDecoder.java:462) > > at org.elasticsearch.common.netty.handler.codec.frame. > > FrameDecoder.callDecode(FrameDecoder.java:443) > > at org.elasticsearch.common.netty.handler.codec.frame. > > FrameDecoder.messageReceived(FrameDecoder.java:303) > > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. > > handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > sendUpstream(DefaultChannelPipeline.java:564) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ > > DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > > 791) > > at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream( > > OpenChannelsHandler.java:74) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > sendUpstream(DefaultChannelPipeline.java:564) > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > sendUpstream(DefaultChannelPipeline.java:559) > > at org.elasticsearch.common.netty.channel.Channels. > > fireMessageReceived(Channels.java:268) > > at org.elasticsearch.common.netty.channel.Channels. > > fireMessageReceived(Channels.java:255) > > at org.elasticsearch.common.netty.channel.socket.nio. > > NioWorker.read(NioWorker.java:88) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioWorker.process(AbstractNioWorker.java:108) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.run(AbstractNioSelector.java:318) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > at org.elasticsearch.common.netty.channel.socket.nio. > > NioWorker.run(NioWorker.java:178) > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > ThreadRenamingRunnable.java:108) > > at org.elasticsearch.common.netty.util.internal. > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > > > This behavior did not occur in 1.1.1. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF0Yx0r1dyi7Lrw0pVsE5v8r-KDgTasmt4j1F-muNoKtg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF0Yx0r1dyi7Lrw0pVsE5v8r-KDgTasmt4j1F-muNoKtg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![schmichael\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/schmichael_2/32/1249_2.png) [@schmichael\_2](https://discuss.elastic.co/u/schmichael_2)\ **Post date:** [September 29, 2014, 10:25pm UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/5 "2014-09-29T22:25:53Z")
Thanks Jorg! I'm marking my thread as complete since the issue already has a better summary of the problem than I've posted here. On Monday, September 29, 2014 3:18:08 PM UTC-7, Jörg Prante wrote: > There is already an issue opened for this: > > [Stats: \_status with #shards \>\> queue capacity failing with BroadcastShardOperationFailedException · Issue #7916 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/7916) > > Jörg > > On Mon, Sep 29, 2014 at 11:52 PM, schmichael \<[michael....@gmail.com](mailto:michael....@gmail.com) > \\> wrote: > > > The behavior does not seem to occur with 1.3.2. > > > > Both kopf and bigdesk cause the error in 1.3.3, and I was running the > > versions of them appropriate for 1.3. > > > > On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote: > > > > > It appears fetching index status information in Elasticsearch 1.3.3 > > > causes logspam and massive CPU util. > > > > > > All of the loglines are of the form: > > > > > > 2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status > > > [local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC\_wwXcjg], [R], > > > s[STARTED]: failed to executed [org.elasticsearch.action. > > > admin.indices.status.IndicesStatusRequest@7cc1c6f4] > > > org.elasticsearch.transport.RemoteTransportException: > > > [remote-alias][inet[/192.168.115.43:9300]][indices/status/s] > > > Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: > > > rejected execution (queue capacity 100) on org.elasticsearch.transport. > > > netty.MessageChannelHandler$RequestHandler@4cd03105 > > > at org.elasticsearch.common.util.concurrent.EsAbortPolicy. > > > rejectedExecution(EsAbortPolicy.java:62) > > > at java.util.concurrent.ThreadPoolExecutor.reject( > > > ThreadPoolExecutor.java:821) > > > at java.util.concurrent.ThreadPoolExecutor.execute( > > > ThreadPoolExecutor.java:1372) > > > at org.elasticsearch.transport.netty.MessageChannelHandler. > > > handleRequest(MessageChannelHandler.java:219) > > > at org.elasticsearch.transport.netty.MessageChannelHandler. > > > messageReceived(MessageChannelHandler.java:111) > > > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. > > > handleUpstream(SimpleChannelUpstreamHandler.java:70) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > > sendUpstream(DefaultChannelPipeline.java:564) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ > > > DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > > > 791) > > > at org.elasticsearch.common.netty.channel.Channels. > > > fireMessageReceived(Channels.java:296) > > > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. > > > unfoldAndFireMessageReceived(FrameDecoder.java:462) > > > at org.elasticsearch.common.netty.handler.codec.frame. > > > FrameDecoder.callDecode(FrameDecoder.java:443) > > > at org.elasticsearch.common.netty.handler.codec.frame. > > > FrameDecoder.messageReceived(FrameDecoder.java:303) > > > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. > > > handleUpstream(SimpleChannelUpstreamHandler.java:70) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > > sendUpstream(DefaultChannelPipeline.java:564) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$ > > > DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java: > > > 791) > > > at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream( > > > OpenChannelsHandler.java:74) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > > sendUpstream(DefaultChannelPipeline.java:564) > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > > > sendUpstream(DefaultChannelPipeline.java:559) > > > at org.elasticsearch.common.netty.channel.Channels. > > > fireMessageReceived(Channels.java:268) > > > at org.elasticsearch.common.netty.channel.Channels. > > > fireMessageReceived(Channels.java:255) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > NioWorker.read(NioWorker.java:88) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioWorker.process(AbstractNioWorker.java:108) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioSelector.run(AbstractNioSelector.java:318) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > NioWorker.run(NioWorker.java:178) > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > > ThreadRenamingRunnable.java:108) > > > at org.elasticsearch.common.netty.util.internal. > > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > This behavior did not occur in 1.1.1. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a682e78d-6454-4397-aae2-d64f12312c8d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a682e78d-6454-4397-aae2-d64f12312c8d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/logspam-caused-by-getting-index-status-in-1-3-3/20006/6 "2017-07-06T00:59:10Z")