Not stable some time

[darkyoung]

Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes
running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the
load average raised.

I checked the ES logs and get the info below, is there something I can do
to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES
server.

[2012-03-04 13:49:29,639][DEBUG][http.netty ] [Wizard] Caught
exception while handling client http traffic, closing connection [id:
0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

[darkyoung]

If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：

Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes
running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the
load average raised.

I checked the ES logs and get the info below, is there something I can do
to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES
server.

[2012-03-04 13:49:29,639][DEBUG][http.netty ] [Wizard]
Caught exception while handling client http traffic, closing connection
[id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

[zhouxiang]

i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒，Sean Wu写道：

If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：

Hi,

I'm using php client Elastica to communicate with ES, and I have two
nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and
the load average raised.

I checked the ES logs and get the info below, is there something I can do
to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES
server.

[2012-03-04 13:49:29,639][DEBUG][http.netty ] [Wizard]
Caught exception while handling client http traffic, closing connection
[id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

[Shairon_Toledo]

I got same issue twice this week, the cluster only get back when all nodes
are restarted. The http port hangs all connection too.
From Netty code I see

void writeFromUserCode(final AbstractNioChannel<?> channel) {
if (!channel.isConnected()) {
cleanUpWriteBuffer(channel);
return;
}

I infer that any node is closing the stream before buffer in the message
event be read or write in transport layer.

I will try reproduce it by ES tests. Meanwhile if you have an idea please
let me know.

ES 0.19.3
Indexer: Java ES client jar 0.19.0
Searcher: rubberband

My log:

[2012-08-23 20:12:37,704][DEBUG][http.netty ]
[prod-search-202] Caught exception while handling client http traffic,
closing connection [id: 0x3fe40812, /10.4.70.11:53934 :>
/10.251.94.255:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)
at
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)
at
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:

i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒，Sean Wu写道：

If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：

Hi,

I'm using php client Elastica to communicate with ES, and I have two
nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and
the load average raised.

I checked the ES logs and get the info below, is there something I can
do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master
ES server.

[2012-03-04 13:49:29,639][DEBUG][http.netty ] [Wizard]
Caught exception while handling client http traffic, closing connection
[id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)
at
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)
at
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)
at
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

--

[Shairon_Toledo]

I took a snapshot via jstack when it happened, the cpu was in 100%

https://gist.github.com/shairontoledo/490f94ba45000bee695d

On Thu, Aug 23, 2012 at 6:02 PM, Shairon Toledo shairon.toledo@gmail.comwrote:

I got same issue twice this week, the cluster only get back when all nodes
are restarted. The http port hangs all connection too.
From Netty code I see

void writeFromUserCode(final AbstractNioChannel<?> channel) {
if (!channel.isConnected()) {
cleanUpWriteBuffer(channel);
return;
}

I infer that any node is closing the stream before buffer in the message
event be read or write in transport layer.

I will try reproduce it by ES tests. Meanwhile if you have an idea please
let me know.

ES 0.19.3
Indexer: Java ES client jar 0.19.0
Searcher: rubberband

My log:

[2012-08-23 20:12:37,704][DEBUG][http.netty ]
[prod-search-202] Caught exception while handling client http traffic,
closing connection [id: 0x3fe40812, /10.4.70.11:53934 :> /
10.251.94.255:9200]
java.nio.channels.ClosedChannelException
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)
at
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)
at
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
at
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)
at
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)
at
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)
at
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)
at
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:

i meet the same problerm.
did you fix it now ?
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒，**Sean Wu写道：

If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒，**Ocean Wu写道：

Hi,

I'm using php client Elastica to communicate with ES, and I have two
nodes running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and
the load average raised.

I checked the ES logs and get the info below, is there something I can
do to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master
ES server.

[2012-03-04 13:49:29,639][DEBUG][http.**netty ] [Wizard]
Caught exception while handling client http traffic, closing connection
[id: 0x477966e4, /192.168.1.6:54837 :> /192.168.1.27:9200]
java.nio.channels.**ClosedChannelException
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.cleanUpWriteBuffer(**NioWorker.java:645)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.writeFromUserCode(**NioWorker.java:372)
at org.elasticsearch.common.netty.channel.socket.nio.
NioServerSocketPipelineSink.handleAcceptedSocket(
NioServerSocketPipelineSink.**java:137)
at org.elasticsearch.common.netty.channel.socket.nio.
NioServerSocketPipelineSink.**eventSunk(*NioServerSocketPipelineSink.
*java:76)
at org.elasticsearch.common.**netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendDownstream(
DefaultChannelPipeline.java:**771)
at org.elasticsearch.common.netty.channel.Channels.write(
Channels.java:632)
at org.elasticsearch.common.netty.handler.codec.oneone.
OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(
Channels.java:611)
at org.elasticsearch.common.netty.channel.Channels.write(
Channels.java:578)
at org.elasticsearch.common.netty.channel.AbstractChannel.
write(AbstractChannel.java:251)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(
NettyHttpChannel.java:156)
at org.elasticsearch.rest.action.search.RestSearchAction$1.
onResponse(RestSearchAction.java:99)
at org.elasticsearch.rest.action.search.RestSearchAction$1.
onResponse(RestSearchAction.java:92)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(
TransportSearchQueryThenFetchAction.java:189)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.finishHim(
TransportSearchQueryThenFetchAction.java:169)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(
TransportSearchQueryThenFetchAction.java:150)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(
TransportSearchQueryThenFetchAction.java:145)
at org.elasticsearch.search.**action.SearchServiceTransportAction$
8.handleResponse(**SearchServiceTransportAction.**java:327)
at org.elasticsearch.search.**action.SearchServiceTransportAction$
8.handleResponse(**SearchServiceTransportAction.**java:320)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleResponse(**MessageChannelHandler.java:**252)
at org.elasticsearch.transport.netty.MessageChannelHandler.
process(MessageChannelHandler.**java:223)
at org.elasticsearch.transport.netty.MessageChannelHandler.
callDecode(**MessageChannelHandler.java:**147)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(**MessageChannelHandler.java:**103)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:80)
at org.elasticsearch.common.**netty.channel.DefaultChannelPipeline.
sendUpstream(**DefaultChannelPipeline.java:**564)
at org.elasticsearch.common.**netty.channel.DefaultChannelPipeline.
sendUpstream(**DefaultChannelPipeline.java:**559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:274)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:261)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:**351)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.processSelectedKeys(**NioWorker.java:282)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:**202)
at org.elasticsearch.common.**netty.util.*ThreadRenamingRunnable.run(
*ThreadRenamingRunnable.java:**108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(**DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.**java:679)

--

--

Shairon Toledo

Microsoft Developer News | C++,C# & .NET Framework & More | CodeGuru

--