ElasticSearch exception: java.lang.IllegalStateException: cannot send more responses than requests

pcdinh · September 27, 2011, 9:51am

Hi,

I got a lot of exceptions in the log file

[2011-09-27 08:39:56,940][WARN ][http.netty ] [Professor X]
Caught exception while handling client http traffic, closing connection
java.lang.IllegalStateException: cannot send more responses than requests
at
org.elasticsearch.common.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:102)
at
org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:266)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexAction.java:123)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexAction.java:101)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:493)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:465)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:377)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Version in use: 0.17.7

What does it mean?

Thanks

kimchy · September 27, 2011, 9:13pm

Are you sure you are using 0.17.7? it basically means that a response was
sent twice on the same open channel, which should not happen.

On Tue, Sep 27, 2011 at 12:51 PM, pcdinh pcdinh@gmail.com wrote:

Hi,

I got a lot of exceptions in the log file

[2011-09-27 08:39:56,940][WARN ][http.netty ] [Professor X]
Caught exception while handling client http traffic, closing connection
java.lang.IllegalStateException: cannot send more responses than requests
at
org.elasticsearch.common.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:102)
at
org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:266)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexAction.java:123)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexAction.java:101)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:493)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:465)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:377)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Version in use: 0.17.7

What does it mean?

Thanks

Josh_Holtzman · November 21, 2011, 5:01pm

I see the same exception under heavy load on elasticsearch 0.18.2.
I'm using the multithreaded apache http client to run performance
tests, so this could have something to do with handling of multiple
concurrent persistent connections:

HttpClient httpClient = new DefaultHttpClient(new
ThreadSafeClientConnManager());

I'll re-run my tests using separate http clients per thread to see if
I can reproduce it.

Josh

On Sep 27, 1:13 pm, Shay Banon kim...@gmail.com wrote:

Are you sure you are using 0.17.7? it basically means that a response was
sent twice on the same open channel, which should not happen.

On Tue, Sep 27, 2011 at 12:51 PM, pcdinh pcd...@gmail.com wrote:

Hi,

I got a lot of exceptions in the log file

[2011-09-27 08:39:56,940][WARN ][http.netty ] [Professor X]
Caught exception while handling client http traffic, closing connection
java.lang.IllegalStateException: cannot send more responses than requests
at
org.elasticsearch.common.netty.handler.codec.http.HttpContentEncoder.writeR equested(HttpContentEncoder.java:102)
at
org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleDownstrea m(SimpleChannelHandler.java:266)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea m(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea m(DefaultChannelPipeline.java:582)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChanne l.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel .java:156)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA ction.java:123)
at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA ction.java:101)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction$AsyncShardOperationAction.performReplicas(TransportShardReplicat ionOperationAction.java:493)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplica tionOperationAction.java:465)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperati onAction.java:377)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908)
at java.lang.Thread.run(Thread.java:662)

Version in use: 0.17.7

What does it mean?

Thanks

kimchy · November 22, 2011, 12:41pm

Can you gist the exception you got?

On Mon, Nov 21, 2011 at 7:01 PM, jholtzman jmholtzman@gmail.com wrote:

I see the same exception under heavy load on elasticsearch 0.18.2.
I'm using the multithreaded apache http client to run performance
tests, so this could have something to do with handling of multiple
concurrent persistent connections:

HttpClient httpClient = new DefaultHttpClient(new
ThreadSafeClientConnManager());

I'll re-run my tests using separate http clients per thread to see if
I can reproduce it.

Josh

On Sep 27, 1:13 pm, Shay Banon kim...@gmail.com wrote:

Are you sure you are using 0.17.7? it basically means that a response was
sent twice on the same open channel, which should not happen.

On Tue, Sep 27, 2011 at 12:51 PM, pcdinh pcd...@gmail.com wrote:

Hi,

I got a lot of exceptions in the log file

[2011-09-27 08:39:56,940][WARN ][http.netty ] [Professor
X]
Caught exception while handling client http traffic, closing connection
java.lang.IllegalStateException: cannot send more responses than
requests
at

org.elasticsearch.common.netty.handler.codec.http.HttpContentEncoder.writeR
equested(HttpContentEncoder.java:102)
    at
org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleDownstrea
m(SimpleChannelHandler.java:266)
    at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea
m(DefaultChannelPipeline.java:591)
    at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea
m(DefaultChannelPipeline.java:582)
    at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
    at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
    at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChanne
l.java:251)
    at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel
.java:156)
    at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA
ction.java:123)
    at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA
ction.java:101)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction.performReplicas(TransportShardReplicat
ionOperationAction.java:493)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplica
tionOperationAction.java:465)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperati
onAction.java:377)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
908)
    at java.lang.Thread.run(Thread.java:662)
Version in use: 0.17.7

What does it mean?

Thanks

kimchy · November 22, 2011, 12:42pm

And, if you have a simple test case, would love to run it myself.

On Tue, Nov 22, 2011 at 2:41 PM, Shay Banon kimchy@gmail.com wrote:

Can you gist the exception you got?

On Mon, Nov 21, 2011 at 7:01 PM, jholtzman jmholtzman@gmail.com wrote:
I see the same exception under heavy load on elasticsearch 0.18.2.
I'm using the multithreaded apache http client to run performance
tests, so this could have something to do with handling of multiple
concurrent persistent connections:

HttpClient httpClient = new DefaultHttpClient(new
ThreadSafeClientConnManager());

I'll re-run my tests using separate http clients per thread to see if
I can reproduce it.

Josh

On Sep 27, 1:13 pm, Shay Banon kim...@gmail.com wrote:

Are you sure you are using 0.17.7? it basically means that a response
was
sent twice on the same open channel, which should not happen.

On Tue, Sep 27, 2011 at 12:51 PM, pcdinh pcd...@gmail.com wrote:

Hi,

I got a lot of exceptions in the log file

[2011-09-27 08:39:56,940][WARN ][http.netty ]
[Professor X]
Caught exception while handling client http traffic, closing
connection
java.lang.IllegalStateException: cannot send more responses than
requests
at

org.elasticsearch.common.netty.handler.codec.http.HttpContentEncoder.writeR
equested(HttpContentEncoder.java:102)
    at
org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleDownstrea
m(SimpleChannelHandler.java:266)
    at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea
m(DefaultChannelPipeline.java:591)
    at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstrea
m(DefaultChannelPipeline.java:582)
    at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
    at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
    at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChanne
l.java:251)
    at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel
.java:156)
    at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA
ction.java:123)
    at
org.elasticsearch.rest.action.index.RestIndexAction$1.onResponse(RestIndexA
ction.java:101)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction.performReplicas(TransportShardReplicat
ionOperationAction.java:493)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplica
tionOperationAction.java:465)
    at
org.elasticsearch.action.support.replication.TransportShardReplicationOpera
tionAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperati
onAction.java:377)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
908)
    at java.lang.Thread.run(Thread.java:662)
Version in use: 0.17.7

What does it mean?

Thanks