Message not fully read

Hello,
I was having this issue and took me around 3 hours to find a solution since
I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:

"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE2][inet[/10.136.56.56:9300]][search/phase/query/id]];
nested: EOFException; "
} ]
},

and in the elasticsearch logs I was getting the next exception:

[2013-05-20 12:58:57,595][DEBUG][action.search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]
Caused by: java.io.EOFException
at
org.elasticsearch.transport.netty.ChannelBufferStreamInput.readBytes(ChannelBufferStreamInput.java:139)
at
org.elasticsearch.common.io.stream.AdapterStreamInput.readBytes(AdapterStreamInput.java:40)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:98)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:90)
at
org.elasticsearch.search.dfs.AggregatedDfs.readFrom(AggregatedDfs.java:78)
at
org.elasticsearch.search.dfs.AggregatedDfs.readAggregatedDfs(AggregatedDfs.java:68)
at
org.elasticsearch.search.query.QuerySearchRequest.readFrom(QuerySearchRequest.java:62)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.SearchServiceTransportAction$1@76a3c7b1, error [false], resetting

The solution was to change the SearchType I was using in my java code from
"DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1 master
(no data) and two data slave nodes.

Kind regards,

Ernesto

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

you are using 0.90 right? We fixed a bug related to 0.90 where DFS was
broken when you upgraded from a 0.20 index. you have 2 options here, one is
to use a non-DFS search type and the other is to optimize your index but
this might still trigger some of those failures. With the next release
0.90.1 (which will come soon) this will go away.

simon

On Monday, May 20, 2013 1:09:53 PM UTC+2, Ernesto Reig wrote:

Hello,
I was having this issue and took me around 3 hours to find a solution
since I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:

"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[Failed to deserialize response
of type [org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
TransportSerializationException[Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" :
"RemoteTransportException[[IPTOOL-ES-NODE2][inet[/10.136.56.56:9300]][search/phase/query/id]];
nested: EOFException; "
} ]
},

and in the elasticsearch logs I was getting the next exception:

[2013-05-20 12:58:57,595][DEBUG][action.search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.136.56.55:9300]][search/phase/query/id]
Caused by: java.io.EOFException
at
org.elasticsearch.transport.netty.ChannelBufferStreamInput.readBytes(ChannelBufferStreamInput.java:139)
at
org.elasticsearch.common.io.stream.AdapterStreamInput.readBytes(AdapterStreamInput.java:40)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:98)
at
org.elasticsearch.common.io.stream.StreamInput.readBytesRef(StreamInput.java:90)
at
org.elasticsearch.search.dfs.AggregatedDfs.readFrom(AggregatedDfs.java:78)
at
org.elasticsearch.search.dfs.AggregatedDfs.readAggregatedDfs(AggregatedDfs.java:68)
at
org.elasticsearch.search.query.QuerySearchRequest.readFrom(QuerySearchRequest.java:62)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:207)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.SearchServiceTransportAction$1@76a3c7b1, error [false],
resetting

The solution was to change the SearchType I was using in my java code from
"DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1
master (no data) and two data slave nodes.

Kind regards,

Ernesto

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello Simon,
yes, I am using 0.90. For now, we´ll use the default search (query then
fetch) until 0.90.1 comes.

Thank you very much,

Ernesto

On 20 May 2013 13:14, simonw simon.willnauer@elasticsearch.com wrote:

you are using 0.90 right? We fixed a bug related to 0.90 where DFS was
broken when you upgraded from a 0.20 index. you have 2 options here, one is
to use a non-DFS search type and the other is to optimize your index but
this might still trigger some of those failures. With the next release
0.90.1 (which will come soon) this will go away.

simon

On Monday, May 20, 2013 1:09:53 PM UTC+2, Ernesto Reig wrote:

Hello,
I was having this issue and took me around 3 hours to find a solution
since I couldn´t find any info related to it. That´s why I am posting this
solution so anybody having the problem can save time.* I would also be
grateful if someone could explain why this happens*.
I was getting in the response from my java client:

"failures" : [ {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationExceptio
n[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IOException[Expected handle header, got [-109]]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationExceptio
n[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IndexOutOfBoundsException[**Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[**Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]];
nested: TransportSerializationExceptio
n[Failed to deserialize response
of type [org.elasticsearch.search.dfs.**DfsSearchResult]]; nested:
IndexOutOfBoundsException[Readable byte limit exceeded: 119]; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[[

IPTOOL-ES-NODE1][inet[/10.136.**56.55:9300]][search/phase/query/id]];
nested: EOFException; "
}, {
"status" : 500,
"reason" : "RemoteTransportException[[

IPTOOL-ES-NODE2][inet[/10.136.**56.56:9300]][search/phase/**query/id]];
nested: EOFException; "
} ]
},

and in the elasticsearch logs I was getting the next exception:

[2013-05-20 12:58:57,595][DEBUG][action.**search.type ]
[IPTOOL-ES-MASTER] [1492] Failed to execute query phase
org.elasticsearch.transport.**RemoteTransportException:
[IPTOOL-ES-NODE1][inet[/10.**136.56.55:9300]][search/phase/**query/id]
Caused by: java.io.EOFException
at org.elasticsearch.transport.netty.
ChannelBufferStreamInput.**readBytes(ChannelBufferStreamInput.java:
139)
at org.elasticsearch.common.io.stream.AdapterStreamInput.
readBytes(AdapterStreamInput.**java:40)
at org.elasticsearch.common.io.stream.StreamInput.
readBytesRef(StreamInput.java:**98)
at org.elasticsearch.common.io.stream.StreamInput.
readBytesRef(StreamInput.java:**90)
at org.elasticsearch.search.dfs.AggregatedDfs.readFrom(
AggregatedDfs.java:78)
at org.elasticsearch.search.dfs.AggregatedDfs.
readAggregatedDfs(**AggregatedDfs.java:68)
at org.elasticsearch.search.query.QuerySearchRequest.
readFrom(QuerySearchRequest.**java:62)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleRequest(**MessageChannelHandler.java:**207)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(**MessageChannelHandler.java:**111)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:70)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$**DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:**791)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:296)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.**unfoldAndFireMessageReceived(**FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.callDecode(**FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.messageReceived(**FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:70)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$**DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:**791)
at org.elasticsearch.common.netty.OpenChannelsHandler.
handleUpstream(**OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline.**sendUpstream(DefaultChannelPipeline.java:
559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:268)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:255)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:**88)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.process(**AbstractNioWorker.java:107)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(**AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(**AbstractNioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:**178)
at org.elasticsearch.common.netty.util.
ThreadRenamingRunnable.run(**ThreadRenamingRunnable.java:**108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(**DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)
[2013-05-20 12:58:57,599][WARN ][transport.netty ]
[IPTOOL-ES-MASTER] Message not fully read (response) for [21689] handler
org.elasticsearch.sea
rch.action.**SearchServiceTransportAction$**1@76a3c7b1, error [false],
resetting

The solution was to change the SearchType I was using in my java code
from "DFS_QUERY_THEN_FETCH" to "DEFAULT".
The configuration I am using is a cluster composed by three nodes, 1
master (no data) and two data slave nodes.

Kind regards,

Ernesto

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RW-wr3h-GNc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.