# Not stable some time

**URL:** <https://discuss.elastic.co/t/not-stable-some-time/6876>\
**Category:** Elasticsearch\
**Created:** [March 4, 2012, 6:05am UTC](https://discuss.elastic.co/t/not-stable-some-time/6876 "2012-03-04T06:05:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![darkyoung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darkyoung/32/2924_2.png) [@darkyoung](https://discuss.elastic.co/u/darkyoung)\
**Post date:** [March 4, 2012, 6:05am UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/1 "2012-03-04T06:05:57Z")

</div>

Hi,

I'm using php client Elastica to communicate with ES, and I have two nodes  
running ES 0.18.7.

Some time the php client will get "Operation timed out" Exception, and the  
load average raised.

I checked the ES logs and get the info below, is there something I can do  
to fix this?

192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES  
server.

[2012-03-04 13:49:29,639][DEBUG][http.netty] [Wizard] Caught  
exception while handling client http traffic, closing connection [id:  
0x477966e4, /192.168.1.6:54837 :\> /192.168.1.27:9200]  
java.nio.channels.ClosedChannelException  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)  
at  
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)  
at  
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)  
at  
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)  
at  
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)  
at  
org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)  
at  
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)  
at  
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)  
at  
org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)  
at  
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
at java.lang.Thread.run(Thread.java:679)

---

<div class="post-metadata">

**Author:** ![darkyoung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darkyoung/32/2924_2.png) [@darkyoung](https://discuss.elastic.co/u/darkyoung)\
**Post date:** [March 4, 2012, 6:23am UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/2 "2012-03-04T06:23:09Z")

</div>

If I restart the server, it will be better for some time, hours or days.

在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：

> Hi,
> 
> I'm using php client Elastica to communicate with ES, and I have two nodes  
> running ES 0.18.7.
> 
> Some time the php client will get "Operation timed out" Exception, and the  
> load average raised.
> 
> I checked the ES logs and get the info below, is there something I can do  
> to fix this?
> 
> 192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES  
> server.
> 
> [2012-03-04 13:49:29,639][DEBUG][http.netty] [Wizard]  
> Caught exception while handling client http traffic, closing connection  
> [id: 0x477966e4, /192.168.1.6:54837 :\> /192.168.1.27:9200]  
> java.nio.channels.ClosedChannelException  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)  
> at  
> org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)  
> at  
> org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)  
> at  
> org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)  
> at  
> org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)  
> at  
> org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)  
> at  
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> at java.lang.Thread.run(Thread.java:679)

---

<div class="post-metadata">

**Author:** ![zhouxiang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zhouxiang/32/2775_2.png) [@zhouxiang](https://discuss.elastic.co/u/zhouxiang)\
**Post date:** [August 10, 2012, 4:03am UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/3 "2012-08-10T04:03:53Z")

</div>

i meet the same problerm.  
did you fix it now ?  
3ks

在 2012年3月4日星期日UTC+8下午2时23分09秒，Sean Wu写道：

> If I restart the server, it will be better for some time, hours or days.
> 
> 在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：
> 
> > Hi,
> > 
> > I'm using php client Elastica to communicate with ES, and I have two  
> > nodes running ES 0.18.7.
> > 
> > Some time the php client will get "Operation timed out" Exception, and  
> > the load average raised.
> > 
> > I checked the ES logs and get the info below, is there something I can do  
> > to fix this?
> > 
> > 192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master ES  
> > server.
> > 
> > [2012-03-04 13:49:29,639][DEBUG][http.netty] [Wizard]  
> > Caught exception while handling client http traffic, closing connection  
> > [id: 0x477966e4, /192.168.1.6:54837 :\> /192.168.1.27:9200]  
> > java.nio.channels.ClosedChannelException  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)  
> > at  
> > org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)  
> > at  
> > org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)  
> > at  
> > org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)  
> > at  
> > org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)  
> > at  
> > org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)  
> > at  
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)  
> > at  
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > at  
> > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> > at java.lang.Thread.run(Thread.java:679)

---

<div class="post-metadata">

**Author:** ![Shairon\_Toledo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shairon_toledo/32/2490_2.png) [@Shairon\_Toledo](https://discuss.elastic.co/u/Shairon_Toledo)\
**Post date:** [August 23, 2012, 9:02pm UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/4 "2012-08-23T21:02:04Z")

</div>

I got same issue twice this week, the cluster only get back when all nodes  
are restarted. The http port hangs all connection too.  
From Netty code I see

void writeFromUserCode(final AbstractNioChannel\<?\> channel) {  
if (!channel.isConnected()) {  
cleanUpWriteBuffer(channel);  
return;  
}

I infer that any node is closing the stream before buffer in the message  
event be read or write in transport layer.

I will try reproduce it by ES tests. Meanwhile if you have an idea please  
let me know.

ES 0.19.3  
Indexer: Java ES client jar 0.19.0  
Searcher: rubberband

My log:

[2012-08-23 20:12:37,704][DEBUG][http.netty]  
[prod-search-202] Caught exception while handling client http traffic,  
closing connection [id: 0x3fe40812, /10.4.70.11:53934 :\>  
/10.251.94.255:9200]  
java.nio.channels.ClosedChannelException  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)  
at  
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)  
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)  
at  
org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)  
at  
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)  
at  
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)  
at  
org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)  
at  
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)  
at  
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)  
at  
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)  
at  
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)  
at  
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)  
at  
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)  
at  
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)

On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:

> i meet the same problerm.  
> did you fix it now ?  
> 3ks
> 
> 在 2012年3月4日星期日UTC+8下午2时23分09秒，Sean Wu写道：
> 
> > If I restart the server, it will be better for some time, hours or days.
> > 
> > 在 2012年3月4日星期日UTC+8下午2时05分57秒，Ocean Wu写道：
> > 
> > > Hi,
> > > 
> > > I'm using php client Elastica to communicate with ES, and I have two  
> > > nodes running ES 0.18.7.
> > > 
> > > Some time the php client will get "Operation timed out" Exception, and  
> > > the load average raised.
> > > 
> > > I checked the ES logs and get the info below, is there something I can  
> > > do to fix this?
> > > 
> > > 192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master  
> > > ES server.
> > > 
> > > [2012-03-04 13:49:29,639][DEBUG][http.netty] [Wizard]  
> > > Caught exception while handling client http traffic, closing connection  
> > > [id: 0x477966e4, /192.168.1.6:54837 :\> /192.168.1.27:9200]  
> > > java.nio.channels.ClosedChannelException  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)  
> > > at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)  
> > > at  
> > > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:632)  
> > > at  
> > > org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)  
> > > at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
> > > at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
> > > at  
> > > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:611)  
> > > at  
> > > org.elasticsearch.common.netty.channel.Channels.write(Channels.java:578)  
> > > at  
> > > org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:251)  
> > > at  
> > > org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:156)  
> > > at  
> > > org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:99)  
> > > at  
> > > org.elasticsearch.rest.action.search.RestSearchAction$1.onResponse(RestSearchAction.java:92)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:189)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:169)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:150)  
> > > at  
> > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:145)  
> > > at  
> > > org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:327)  
> > > at  
> > > org.elasticsearch.search.action.SearchServiceTransportAction$8.handleResponse(SearchServiceTransportAction.java:320)  
> > > at  
> > > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:252)  
> > > at  
> > > org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:223)  
> > > at  
> > > org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:147)  
> > > at  
> > > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)  
> > > at  
> > > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> > > at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
> > > at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
> > > at  
> > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> > > at  
> > > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)  
> > > at  
> > > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)  
> > > at  
> > > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > > at  
> > > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> > > at java.lang.Thread.run(Thread.java:679)

--

---

<div class="post-metadata">

**Author:** ![Shairon\_Toledo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shairon_toledo/32/2490_2.png) [@Shairon\_Toledo](https://discuss.elastic.co/u/Shairon_Toledo)\
**Post date:** [August 24, 2012, 3:40pm UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/5 "2012-08-24T15:40:14Z")

</div>

I took a snapshot via jstack when it happened, the cpu was in 100%

> <https://gist.github.com/shairontoledo/490f94ba45000bee695d>

On Thu, Aug 23, 2012 at 6:02 PM, Shairon Toledo [shairon.toledo@gmail.com](mailto:shairon.toledo@gmail.com)wrote:

> I got same issue twice this week, the cluster only get back when all nodes  
> are restarted. The http port hangs all connection too.  
> From Netty code I see
> 
> void writeFromUserCode(final AbstractNioChannel\<?\> channel) {  
> if (!channel.isConnected()) {  
> cleanUpWriteBuffer(channel);  
> return;  
> }
> 
> I infer that any node is closing the stream before buffer in the message  
> event be read or write in transport layer.
> 
> I will try reproduce it by ES tests. Meanwhile if you have an idea please  
> let me know.
> 
> ES 0.19.3  
> Indexer: Java ES client jar 0.19.0  
> Searcher: rubberband
> 
> My log:
> 
> [2012-08-23 20:12:37,704][DEBUG][http.netty]  
> [prod-search-202] Caught exception while handling client http traffic,  
> closing connection [id: 0x3fe40812, /10.4.70.11:53934 :\> /  
> 10.251.94.255:9200]  
> java.nio.channels.ClosedChannelException  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:673)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:400)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:120)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:59)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:780)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:733)  
> at  
> org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:65)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:712)  
> at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:679)  
> at  
> org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:245)  
> at  
> org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:150)  
> at  
> org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:116)  
> at  
> org.elasticsearch.rest.action.admin.cluster.node.info.RestNodesInfoAction$1.onResponse(RestNodesInfoAction.java:106)  
> at  
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:220)  
> at  
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)  
> at  
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)  
> at  
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:180)  
> at  
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleResponse(TransportNodesOperationAction.java:172)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:262)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:233)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:141)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:95)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)
> 
> On Friday, August 10, 2012 1:03:53 AM UTC-3, shinezhou wrote:
> 
> > i meet the same problerm.  
> > did you fix it now ?  
> > 3ks
> > 
> > 在 2012年3月4日星期日UTC+8下午2时23分09秒，\*\*Sean Wu写道：
> > 
> > > If I restart the server, it will be better for some time, hours or days.
> > > 
> > > 在 2012年3月4日星期日UTC+8下午2时05分57秒，\*\*Ocean Wu写道：
> > > 
> > > > Hi,
> > > > 
> > > > I'm using php client Elastica to communicate with ES, and I have two  
> > > > nodes running ES 0.18.7.
> > > > 
> > > > Some time the php client will get "Operation timed out" Exception, and  
> > > > the load average raised.
> > > > 
> > > > I checked the ES logs and get the info below, is there something I can  
> > > > do to fix this?
> > > > 
> > > > 192.168.1.6 is the machine running haproxy, 192.168.1.27 is the master  
> > > > ES server.
> > > > 
> > > > [2012-03-04 13:49:29,639][DEBUG][http.\*\*netty] [Wizard]  
> > > > Caught exception while handling client http traffic, closing connection  
> > > > [id: 0x477966e4, /192.168.1.6:54837 :\> /192.168.1.27:9200]  
> > > > java.nio.channels.\*\*ClosedChannelException  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioWorker.cleanUpWriteBuffer(\*\*NioWorker.java:645)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioWorker.writeFromUserCode(\*\*NioWorker.java:372)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioServerSocketPipelineSink.**handleAcceptedSocket(**  
> > > > NioServerSocketPipelineSink.\*\*java:137)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioServerSocketPipelineSink.\*\*eventSunk(\*_NioServerSocketPipelineSink._  
> > > > \*java:76)  
> > > > at org.elasticsearch.common.\*\*netty.channel. **DefaultChannelPipeline$**  
> > > > DefaultChannelHandlerContext.**sendDownstream(**  
> > > > DefaultChannelPipeline.java:\*\*771)  
> > > > at org.elasticsearch.common.**netty.channel.Channels.write(**  
> > > > Channels.java:632)  
> > > > at org.elasticsearch.common. **netty.handler.codec.oneone.**  
> > > > OneToOneEncoder.**handleDownstream(OneToOneEncoder.java:70)  
> > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.  
> > > > sendDownstream(DefaultChannelPipeline.java:591)  
> > > > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.  
> > > > sendDownstream(DefaultChannelPipeline.java:582)  
> > > > at org.elasticsearch.common.netty.channel.Channels.write(  
> > > > Channels.java:611)  
> > > > at org.elasticsearch.common.netty.channel.Channels.write(  
> > > > Channels.java:578)  
> > > > at org.elasticsearch.common.netty.channel.AbstractChannel.  
> > > > write(AbstractChannel.java:251)  
> > > > at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(  
> > > > NettyHttpChannel.java:156)  
> > > > at org.elasticsearch.rest.action.search.RestSearchAction$1.  
> > > > onResponse(RestSearchAction.java:99)  
> > > > at org.elasticsearch.rest.action.search.RestSearchAction$1.  
> > > > onResponse(RestSearchAction.java:92)  
> > > > at org.elasticsearch.action.search.type.  
> > > > TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(  
> > > > TransportSearchQueryThenFetchAction.java:189)  
> > > > at org.elasticsearch.action.search.type.  
> > > > TransportSearchQueryThenFetchAction$AsyncAction.finishHim(  
> > > > TransportSearchQueryThenFetchAction.java:169)  
> > > > at org.elasticsearch.action.search.type.  
> > > > TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(  
> > > > TransportSearchQueryThenFetchAction.java:150)  
> > > > at org.elasticsearch.action.search.type.  
> > > > TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(  
> > > > TransportSearchQueryThenFetchA**ction.java:145)  
> > > > at org.elasticsearch.search.\*\*action. **SearchServiceTransportAction$**  
> > > > 8.handleResponse(\*\*SearchServiceTransportAction.\*\*java:327)  
> > > > at org.elasticsearch.search.\*\*action. **SearchServiceTransportAction$**  
> > > > 8.handleResponse(\*\*SearchServiceTransportAction.\*\*java:320)  
> > > > at org.elasticsearch.transport. **netty.MessageChannelHandler.**  
> > > > handleResponse(\*\*MessageChannelHandler.java:\*\*252)  
> > > > at org.elasticsearch.transport. **netty.MessageChannelHandler.**  
> > > > process(MessageChannelHandler.\*\*java:223)  
> > > > at org.elasticsearch.transport. **netty.MessageChannelHandler.**  
> > > > callDecode(\*\*MessageChannelHandler.java:\*\*147)  
> > > > at org.elasticsearch.transport. **netty.MessageChannelHandler.**  
> > > > messageReceived(\*\*MessageChannelHandler.java:\*\*103)  
> > > > at org.elasticsearch.common. **netty.channel.**  
> > > > SimpleChannelUpstreamHandler.**handleUpstream(**  
> > > > SimpleChannelUpstreamHandler.\*\*java:80)  
> > > > at org.elasticsearch.common.\*\*netty.channel. **DefaultChannelPipeline.**  
> > > > sendUpstream(\*\*DefaultChannelPipeline.java:\*\*564)  
> > > > at org.elasticsearch.common.\*\*netty.channel. **DefaultChannelPipeline.**  
> > > > sendUpstream(\*\*DefaultChannelPipeline.java:\*\*559)  
> > > > at org.elasticsearch.common. **netty.channel.Channels.**  
> > > > fireMessageReceived(Channels.\*\*java:274)  
> > > > at org.elasticsearch.common. **netty.channel.Channels.**  
> > > > fireMessageReceived(Channels.\*\*java:261)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioWorker.read(NioWorker.java:\*\*351)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioWorker.processSelectedKeys(\*\*NioWorker.java:282)  
> > > > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > > > NioWorker.run(NioWorker.java:\*\*202)  
> > > > at org.elasticsearch.common.\*\*netty.util.\*_ThreadRenamingRunnable.run(_  
> > > > \*ThreadRenamingRunnable.java:\*\*108)  
> > > > at org.elasticsearch.common. **netty.util.internal.**  
> > > > DeadLockProofWorker$1.run(\*\*DeadLockProofWorker.java:44)  
> > > > at java.util.concurrent.**ThreadPoolExecutor.runWorker(**  
> > > > ThreadPoolExecutor.java:1110)  
> > > > at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**  
> > > > ThreadPoolExecutor.java:603)  
> > > > at java.lang.Thread.run(Thread.\*\*java:679)
> > > 
> > > --

--

Shairon Toledo

> **[Microsoft Developer News | C++,C# & .NET Framework & More | CodeGuru](http://hashcode.me)**
>
> 您是否想要找关于幸运飞行艇 官方开奖历史记录 168飞艇全国统一开奖数据 还有幸运飞行艇官方开奖现场直播 幸运168飞艇全天精准稳定计划 这里都能一一提供。CodeGuru is where developers come to share ideas, articles, questions, answers, tips, tricks, comments, downloads, and more related to programming in areas including C++,...

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:15am UTC](https://discuss.elastic.co/t/not-stable-some-time/6876/6 "2017-07-06T03:15:10Z")

</div>