Strange Failure

Mustafa_Sener · July 4, 2012, 8:29am

Hi,
We currently use ES 0.19.0 release. We have a cluster with two nodes. Nodes
are located on two different virtual machines which are on the same
physical server. Yesterday, we had an DS outage. There were no OutOfMemory
exceptions. Both nodes continue to work individually. When I checked logs
of both DS servers, I found following exceptions:

gist.github.com

https://gist.github.com/enguzekli/3045973

Exceptions from DS1 Log files

[2012-07-03 11:35:57,621][INFO ][node                     ] [Xandu_71f67394-f394-4f56-bf3a-de677bb8099b] {0.19.0}[16798]: stopping ...
[2012-07-03 11:35:58,939][DEBUG][action.search.type       ] [Xandu_71f67394-f394-4f56-bf3a-de677bb8099b] [226577095] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [226577095]
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:451)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:424)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:344)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

This file has been truncated. show original

Exceptions from DS2 Log files

[2012-07-03 11:35:59,205][WARN ][transport.netty          ] [Comet_a87ee4a4-ef1e-46b8-b894-5adbd4350e7a] Exception caught on netty layer [[id: 0x3e0bbd1b, /172.27.30.214:54842 => /172.27.30.215:9300]]
java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.elasticsearch.common.netty.buffer.HeapChannelBuffer.getBytes(HeapChannelBuffer.java:111)
        at org.elasticsearch.common.netty.buffer.TruncatedChannelBuffer.getBytes(TruncatedChannelBuffer.java:182)
        at org.elasticsearch.common.netty.channel.socket.oio.OioWorker.write(OioWorker.java:148)
        at org.elasticsearch.common.netty.channel.socket.oio.OioServerSocketPipelineSink.handleAcceptedSocket(OioServerSocketPipelineSink.java:118)
        at org.elasticsearch.common.netty.channel.socket.oio.OioServerSocketPipelineSink.eventSunk(OioServerSocketPipelineSink.java:56)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)

This file has been truncated. show original

Also, I found a bug in one of our processes. This bug causes index
creation/deletion every minute and this situation continued for 4 hours
before DS restart. This means that 240 index creation/deletion operation
performed before this problem occurred. May this cause the node failure?

any ideas about this problem?

Mustafa Sener
www.ifountain.com

Topic		Replies	Views
SearchContextMissingException and out-of-memory crash Elasticsearch	8	472	July 6, 2017
Cluster went to red stage due to SearchContextMissingException Elasticsearch	2	392	July 6, 2017
Node failures Elasticsearch	6	557	July 6, 2017
SearchContextMissingException Failed to execute fetch phase Elasticsearch	2	601	July 6, 2017
Random search errors from one node of three on a previously working cluster Elasticsearch	10	339	July 6, 2017

Strange Failure

any ideas about this problem?

Related topics