Logspam caused by getting index status in 1.3.3

schmichael_2 · September 29, 2014, 9:43pm

It appears fetching index status information in Elasticsearch 1.3.3 causes
logspam and massive CPU util.

All of the loglines are of the form:

2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status
[local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC_wwXcjg], [R],
s[STARTED]: failed to executed
[org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4]
org.elasticsearch.transport.RemoteTransportException:
[remote-alias][inet[/192.168.115.43:9300]][indices/status/s]
Caused by:
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 100) on
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This behavior did not occur in 1.1.1.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

nik9000 · September 29, 2014, 9:50pm

Mind running hot_threads and looking for something telling? That exception
is more describing the symptom then the problem.

On Mon, Sep 29, 2014 at 5:43 PM, schmichael michael.schurter@gmail.com
wrote:

It appears fetching index status information in Elasticsearch 1.3.3 causes
logspam and massive CPU util.

All of the loglines are of the form:

2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status
[local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC_wwXcjg], [R],
s[STARTED]: failed to executed
[org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4
]
org.elasticsearch.transport.RemoteTransportException:
[remote-alias][inet[/192.168.115.43:9300]][indices/status/s]
Caused by:
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 100) on
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This behavior did not occur in 1.1.1.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/365a0bf8-5673-42b1-b623-a3a2574ddcbf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3xhnfOrzCRGa7B3dxotE_%2BeerU8gij9RAk05tw__aajg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

schmichael_2 · September 29, 2014, 9:52pm

The behavior does not seem to occur with 1.3.2.

Both kopf and bigdesk cause the error in 1.3.3, and I was running the
versions of them appropriate for 1.3.

On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote:

It appears fetching index status information in Elasticsearch 1.3.3 causes
logspam and massive CPU util.

All of the loglines are of the form:

2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status
[local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC_wwXcjg], [R],
s[STARTED]: failed to executed
[org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@7cc1c6f4]
org.elasticsearch.transport.RemoteTransportException:
[remote-alias][inet[/192.168.115.43:9300]][indices/status/s]
Caused by:
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 100) on
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@4cd03105
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:219)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This behavior did not occur in 1.1.1.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · September 29, 2014, 10:18pm

There is already an issue opened for this:

github.com/elastic/elasticsearch

Stats: _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException

opened 05:08PM - 29 Sep 14 UTC

closed 09:37AM - 30 Sep 14 UTC

alexbrasetvik

>bug v2.0.0-beta1 v1.4.0.Beta1 v1.5.0 v1.3.4 :Data Management/Stats

Hi. While /_status is deprecated and should be replaced with /_recovery, there …seems to be a regression between 1.3.2 and 1.3.3 when a cluster has lots of shards. The /_status-request fails with e.g: ``` "failures" : [ { "index" : "foo-0104", "shard" : 0, "reason" : "BroadcastShardOperationFailedException[[foo-0104][0] ]; nested: EsRejectedExecutionException[rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c]; " } ] } ``` Elasticsearch logs ``` [2014-09-29 19:05:09,657][DEBUG][action.admin.indices.status] [Sputnik] [foo-0104][0], node[LWIGdaF4QdWwLEz2peydqQ], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@746ef917] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:166) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.start(TransportBroadcastOperationAction.java:150) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:70) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:65) at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:71) at org.elasticsearch.client.support.AbstractIndicesAdminClient.status(AbstractIndicesAdminClient.java:429) at org.elasticsearch.rest.action.admin.indices.status.RestIndicesStatusAction.handleRequest(RestIndicesStatusAction.java:62) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:66) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:177) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:160) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:301) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) ``` This reliably reproduces the issue with a clean install of 1.3.3: - Create 200 indexes, e.g. `>>> for i in range(0, 200): requests.put('http://localhost:9200/foo-%04i' % i, data='{"index": {"number_of_shards": 1, "number_of_replicas": 0}}')` - `curl localhost:9200/_status` This, in turn, causes dashboards like Kopf and Head to break.

Jörg

On Mon, Sep 29, 2014 at 11:52 PM, schmichael michael.schurter@gmail.com
wrote:

The behavior does not seem to occur with 1.3.2.

Both kopf and bigdesk cause the error in 1.3.3, and I was running the
versions of them appropriate for 1.3.

On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote:

It appears fetching index status information in Elasticsearch 1.3.3
causes logspam and massive CPU util.

All of the loglines are of the form:

2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status
[local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC_wwXcjg], [R],
s[STARTED]: failed to executed [org.elasticsearch.action.
admin.indices.status.IndicesStatusRequest@7cc1c6f4]
org.elasticsearch.transport.RemoteTransportException:
[remote-alias][inet[/192.168.115.43:9300]][indices/status/s]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 100) on org.elasticsearch.transport.
netty.MessageChannelHandler$RequestHandler@4cd03105
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.
rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(
ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(
ThreadPoolExecutor.java:1372)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(
MessageChannelHandler.java:219)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(MessageChannelHandler.java:111)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(
OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This behavior did not occur in 1.1.1.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF0Yx0r1dyi7Lrw0pVsE5v8r-KDgTasmt4j1F-muNoKtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

schmichael_2 · September 29, 2014, 10:25pm

Thanks Jorg! I'm marking my thread as complete since the issue already has
a better summary of the problem than I've posted here.

On Monday, September 29, 2014 3:18:08 PM UTC-7, Jörg Prante wrote:

There is already an issue opened for this:

Stats: _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException · Issue #7916 · elastic/elasticsearch · GitHub

Jörg

On Mon, Sep 29, 2014 at 11:52 PM, schmichael <michael....@gmail.com
<javascript:>> wrote:

The behavior does not seem to occur with 1.3.2.

Both kopf and bigdesk cause the error in 1.3.3, and I was running the
versions of them appropriate for 1.3.

On Monday, September 29, 2014 2:43:00 PM UTC-7, schmichael wrote:

It appears fetching index status information in Elasticsearch 1.3.3
causes logspam and massive CPU util.

All of the loglines are of the form:

2014/09/29 21:38:56.663000 [DEBUG] action.admin.indices.status
[local-alias] [logstash-2014.09.29][4], node[KlgByctJTyiO7iC_wwXcjg], [R],
s[STARTED]: failed to executed [org.elasticsearch.action.
admin.indices.status.IndicesStatusRequest@7cc1c6f4]
org.elasticsearch.transport.RemoteTransportException:
[remote-alias][inet[/192.168.115.43:9300]][indices/status/s]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 100) on org.elasticsearch.transport.
netty.MessageChannelHandler$RequestHandler@4cd03105
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.
rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(
ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(
ThreadPoolExecutor.java:1372)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleRequest(MessageChannelHandler.java:219)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(MessageChannelHandler.java:111)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(
OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This behavior did not occur in 1.1.1.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/75b7e353-fe31-4629-ae21-a89f63551798%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a682e78d-6454-4397-aae2-d64f12312c8d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.