Hello,
I was having this issue and took me around 3 hours to find a solution since
I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:
"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE2][inet[/10.136.56.56:9300]][search/phase/query/id]];
nested: EOFException; "
} ]
},
and in the elasticsearch logs I was getting the next exception:
[2013-05-20 12:58:57,595][DEBUG][action.search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]
Caused by: java.io.EOFException
at
org.elasticsearch.transport.netty.ChannelBufferStreamInput.readBytes(ChannelBufferStreamInput.java:139)
at
org.elasticsearch.common.io.stream.AdapterStreamInput.readBytes(AdapterStreamInput.java:40)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:98)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:90)
at
org.elasticsearch.search.dfs.AggregatedDfs.readFrom(AggregatedDfs.java:78)
at
org.elasticsearch.search.dfs.AggregatedDfs.readAggregatedDfs(AggregatedDfs.java:68)
at
org.elasticsearch.search.query.QuerySearchRequest.readFrom(QuerySearchRequest.java:62)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.SearchServiceTransportAction$1@76a3c7b1, error [false], resetting
The solution was to change the SearchType I was using in my java code from
"DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1 master
(no data) and two data slave nodes.
you are using 0.90 right? We fixed a bug related to 0.90 where DFS was
broken when you upgraded from a 0.20 index. you have 2 options here, one is
to use a non-DFS search type and the other is to optimize your index but
this might still trigger some of those failures. With the next release
0.90.1 (which will come soon) this will go away.
simon
On Monday, May 20, 2013 1:09:53 PM UTC+2, Ernesto Reig wrote:
Hello,
I was having this issue and took me around 3 hours to find a solution
since I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:
"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE2][inet[/10.136.56.56:9300]][search/phase/query/id]];
nested: EOFException; "
} ]
},
and in the elasticsearch logs I was getting the next exception:
[2013-05-20 12:58:57,595][DEBUG][action.search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]
Caused by: java.io.EOFException
at
org.elasticsearch.transport.netty.ChannelBufferStreamInput.readBytes(ChannelBufferStreamInput.java:139)
at
org.elasticsearch.common.io.stream.AdapterStreamInput.readBytes(AdapterStreamInput.java:40)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:98)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:90)
at
org.elasticsearch.search.dfs.AggregatedDfs.readFrom(AggregatedDfs.java:78)
at
org.elasticsearch.search.dfs.AggregatedDfs.readAggregatedDfs(AggregatedDfs.java:68)
at
org.elasticsearch.search.query.QuerySearchRequest.readFrom(QuerySearchRequest.java:62)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.SearchServiceTransportAction$1@76a3c7b1, error [false],
resetting
The solution was to change the SearchType I was using in my java code from
"DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1
master (no data) and two data slave nodes.
you are using 0.90 right? We fixed a bug related to 0.90 where DFS was
broken when you upgraded from a 0.20 index. you have 2 options here, one is
to use a non-DFS search type and the other is to optimize your index but
this might still trigger some of those failures. With the next release
0.90.1 (which will come soon) this will go away.
simon
On Monday, May 20, 2013 1:09:53 PM UTC+2, Ernesto Reig wrote:
Hello,
I was having this issue and took me around 3 hours to find a solution
since I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:
"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IndexOutOfBoundsException[**Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[[
IPTOOL-ES-NODE1][inet[/10.136.**56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[[
IPTOOL-ES-NODE2][inet[/10.136.**56.56:9300]][search/phase/**query/id]];
nested: EOFException; "
} ]
},
and in the elasticsearch logs I was getting the next exception:
[2013-05-20 12:58:57,595][DEBUG][action.**search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.**RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.**136.56.55:9300]][search/phase/**query/id]
Caused by: java.io.EOFException
at org.elasticsearch.transport.netty.
ChannelBufferStreamInput.**readBytes(ChannelBufferStreamInput.java:
139)
at org.elasticsearch.common.io.stream.AdapterStreamInput.
readBytes(AdapterStreamInput.**java:40)
at org.elasticsearch.common.io.stream.StreamInput.
readBytesRef(StreamInput.java:**98)
at org.elasticsearch.common.io.stream.StreamInput.
readBytesRef(StreamInput.java:**90)
at org.elasticsearch.search.dfs.AggregatedDfs.readFrom(
AggregatedDfs.java:78)
at org.elasticsearch.search.dfs.AggregatedDfs.
readAggregatedDfs(**AggregatedDfs.java:68)
at org.elasticsearch.search.query.QuerySearchRequest.
readFrom(QuerySearchRequest.**java:62)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleRequest(**MessageChannelHandler.java:**207)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(**MessageChannelHandler.java:**111)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:70)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$**DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:**791)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:296)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.**unfoldAndFireMessageReceived(**FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.callDecode(**FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.messageReceived(**FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:70)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$**DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:**791)
at org.elasticsearch.common.netty.OpenChannelsHandler.
handleUpstream(**OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:268)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:255)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:**88)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.process(**AbstractNioWorker.java:107)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(**AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(**AbstractNioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:**178)
at org.elasticsearch.common.netty.util.
ThreadRenamingRunnable.run(**ThreadRenamingRunnable.java:**108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(**DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.**SearchServiceTransportAction$**1@76a3c7b1, error [false],
resetting
The solution was to change the SearchType I was using in my java code
from "DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1
master (no data) and two data slave nodes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.