Message not fully read (response) for SearchServiceTransportAction issue after upgrading to 0.90.0

keung2640 · May 9, 2013, 10:33am

Hi All,

Sorry for this noob question.

My index has 3 shards and 1 replica. There are 3 nodes(es version 0.90.0).
When I do the query searching with search type DFS_QUERY_THEN_FETCH on some
special fields,
there is a warning 'Message not fully read (response)'. Some of the results
are missing.
The results for subsequent requests keep changing. I can see that there are
3 versions for the search result.

Server log are as followings :
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][1] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][0] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:44:50,568][INFO ][action.admin.cluster.node.shutdown] [Grand
Director] shutting down in [200ms]
[2013-05-09 03:44:50,771][INFO ][action.admin.cluster.node.shutdown] [Grand
Director] initiating requested shutdown...
[2013-05-09 03:44:50,771][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopping ...
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopped
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closing ...
[2013-05-09 03:44:50,833][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closed
[2013-05-09 03:45:35,022][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initializing ...
[2013-05-09 03:45:35,031][INFO ][plugins ] [Baron
Strucker] loaded [], sites []
[2013-05-09 03:45:37,691][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initialized
[2013-05-09 03:45:37,692][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: starting ...
[2013-05-09 03:45:37,828][INFO ][transport ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/10.1.4.220:9300]}
[2013-05-09 03:45:40,933][INFO ][cluster.service ] [Baron
Strucker] detected_master [Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]], added
{[Puishannt][9ogJ5QcyTHiWhMe-mJCPkQ][inet[/10.1.4.221:9300]],[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]],}, reason:
zen-disco-receive(from master [[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]]])
[2013-05-09 03:45:41,031][INFO ][discovery ] [Baron
Strucker] _testinges.0.90.0/THfT5ZzSRPai-PV9bLaP_Q
[2013-05-09 03:45:41,042][INFO ][http ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9200]}, publish_address
{inet[/10.1.4.220:9200]}
[2013-05-09 03:45:41,042][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: started
[2013-05-09 03:46:18,292][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [68] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@28fa1b85,
error [false], resetting
[2013-05-09 03:46:18,304][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [67] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@1441e2aa,
error [false], resetting
[2013-05-09 03:46:18,391][DEBUG][action.search.type ] [Baron
Strucker] [i_product_ys_5][2], node[EtCRx4NOTwuI3bHAn8IGRg], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@71881149]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type [org.elasticsearch.search.dfs.DfsSearchResult]
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:150)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Expected handle header, got [8]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.search.dfs.DfsSearchResult.readFrom(DfsSearchResult.java:140)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)
... 23 more
[2013-05-09 03:46:18,399][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [70] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@326d234c,
error [false], resetting

I found this issue after upgrading the elasticsearch version from 0.19.10
to 0.90.0.
I have no idea why this happens in 0.90.0.
I've verified that only multiple shards will have this problem. Single
shard works fine.
Also, it happens only for my client with JAVA API.

I've tried to re-produce this issue at

gist.github.com

https://gist.github.com/keung2640/5546721

es client with JAVA API

        // init es client
        Settings settings = ImmutableSettings.settingsBuilder()
                .put("cluster.name", "shooting.dfs")
                .put("client.transport.sniff", true)
                .build();
        
        TransportClient client = new TransportClient(settings);
        
        client.addTransportAddress(new InetSocketTransportAddress(
                "localhost" /* hostname */, 9300 /* port */));

This file has been truncated. show original

es server settings

Server settings:
starting 3 nodes(version 0.90.0) with cluster name 'shooting.dfs'.

Create index : 
curl -XPUT 'http://localhost:9200/product/' -d '{
    "settings" : {
        "index" : {
            "number_of_shards" : 3,
            "number_of_replicas" : 1
        },

This file has been truncated. show original

For my client and server setting,
The client es jar version is :

org.elasticsearch
elasticsearch
0.90.0

The es servers are using 0.90.0.
ref : http://www.elasticsearch.org/downloads/0-90-0/
All the java versions on the clusters are the same.

Can I ask is there any problem for my settings?

Many thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · May 9, 2013, 7:06pm

Have you seen this issue?

github.com/elastic/elasticsearch

DFS modes can cause undefined behaviour in 0.90

opened 04:07PM - 08 May 13 UTC

closed 07:55PM - 08 May 13 UTC

s1monw

>bug v0.90.1 v1.0.0.Beta1

Currently some of the transport protocols are broken if Term/FieldStatistics don…'t provide all optional information. For instance we use `totalTermFrequency` in the TermStats but if you omit the term frequency the `totalTermFrequency` is a negative 1 (`-1`). This also happens if an index gets upgrade to 0.90 from a 0.20 version that runs lucene 3.6 and doesn't have these stats so they are -1 by default. This seems to be the cause for #3008 as well as #2932 which eventually fail with NPE since the reading behavior of vlongs is undefined.

Very likely it is related.

Cheers,

Ivan

On Thu, May 9, 2013 at 3:33 AM, keung2640@gmail.com wrote:

Hi All,

Sorry for this noob question.

My index has 3 shards and 1 replica. There are 3 nodes(es version 0.90.0).
When I do the query searching with search type DFS_QUERY_THEN_FETCH on
some special fields,
there is a warning 'Message not fully read (response)'. Some of the
results are missing.
The results for subsequent requests keep changing. I can see that there
are 3 versions for the search result.

Server log are as followings :
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][1] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][0] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:44:50,568][INFO ][action.admin.cluster.node.shutdown]
[Grand Director] shutting down in [200ms]
[2013-05-09 03:44:50,771][INFO ][action.admin.cluster.node.shutdown]
[Grand Director] initiating requested shutdown...
[2013-05-09 03:44:50,771][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopping ...
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopped
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closing ...
[2013-05-09 03:44:50,833][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closed
[2013-05-09 03:45:35,022][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initializing ...
[2013-05-09 03:45:35,031][INFO ][plugins ] [Baron
Strucker] loaded , sites
[2013-05-09 03:45:37,691][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initialized
[2013-05-09 03:45:37,692][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: starting ...
[2013-05-09 03:45:37,828][INFO ][transport ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
10.1.4.220:9300]}
[2013-05-09 03:45:40,933][INFO ][cluster.service ] [Baron
Strucker] detected_master [Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]], added
{[Puishannt][9ogJ5QcyTHiWhMe-mJCPkQ][inet[/10.1.4.221:9300]],[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]],}, reason:
zen-disco-receive(from master [[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]]])
[2013-05-09 03:45:41,031][INFO ][discovery ] [Baron
Strucker] _testinges.0.90.0/THfT5ZzSRPai-PV9bLaP_Q
[2013-05-09 03:45:41,042][INFO ][http ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
10.1.4.220:9200]}
[2013-05-09 03:45:41,042][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: started
[2013-05-09 03:46:18,292][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [68] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@28fa1b85,
error [false], resetting
[2013-05-09 03:46:18,304][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [67] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@1441e2aa,
error [false], resetting
[2013-05-09 03:46:18,391][DEBUG][action.search.type ] [Baron
Strucker] [i_product_ys_5][2], node[EtCRx4NOTwuI3bHAn8IGRg], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@71881149]
org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize response of type [org.elasticsearch.search.dfs.DfsSearchResult]
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:150)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Expected handle header, got [8]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.search.dfs.DfsSearchResult.readFrom(DfsSearchResult.java:140)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)
... 23 more
[2013-05-09 03:46:18,399][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [70] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@326d234c,
error [false], resetting

I found this issue after upgrading the elasticsearch version from 0.19.10
to 0.90.0.
I have no idea why this happens in 0.90.0.
I've verified that only multiple shards will have this problem. Single
shard works fine.
Also, it happens only for my client with JAVA API.

I've tried to re-produce this issue at
Shooting Message not fully read (response) for handler org.elasticsearch.search.action.SearchServiceTransportAction · GitHub

For my client and server setting,
The client es jar version is :

org.elasticsearch
elasticsearch
0.90.0

The es servers are using 0.90.0.
ref : Elasticsearch Platform — Find real-time answers at scale | Elastic
All the java versions on the clusters are the same.

Can I ask is there any problem for my settings?

Many thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

keung2640 · May 10, 2013, 1:04am

Hi Ivan,

Thanks for your info and reference. Let me take a look at it.

Ivan Brusic於 2013年5月10日星期五UTC+8上午3時06分20秒寫道：

Have you seen this issue?
DFS modes can cause undefined behaviour in 0.90 · Issue #3012 · elastic/elasticsearch · GitHub

Very likely it is related.

Cheers,

Ivan

On Thu, May 9, 2013 at 3:33 AM, <keun...@gmail.com <javascript:>> wrote:

Hi All,

Sorry for this noob question.

My index has 3 shards and 1 replica. There are 3 nodes(es version
0.90.0).
When I do the query searching with search type DFS_QUERY_THEN_FETCH on
some special fields,
there is a warning 'Message not fully read (response)'. Some of the
results are missing.
The results for subsequent requests keep changing. I can see that there
are 3 versions for the search result.

Server log are as followings :
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][1] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:43:13,489][INFO ][index.shard.service ] [Grand
Director] [i_product_ywk_5][0] updating refresh_interval from [-1] to [1s]
[2013-05-09 03:44:50,568][INFO ][action.admin.cluster.node.shutdown]
[Grand Director] shutting down in [200ms]
[2013-05-09 03:44:50,771][INFO ][action.admin.cluster.node.shutdown]
[Grand Director] initiating requested shutdown...
[2013-05-09 03:44:50,771][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopping ...
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: stopped
[2013-05-09 03:44:50,823][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closing ...
[2013-05-09 03:44:50,833][INFO ][node ] [Grand
Director] {0.90.0}[8128]: closed
[2013-05-09 03:45:35,022][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initializing ...
[2013-05-09 03:45:35,031][INFO ][plugins ] [Baron
Strucker] loaded , sites
[2013-05-09 03:45:37,691][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: initialized
[2013-05-09 03:45:37,692][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: starting ...
[2013-05-09 03:45:37,828][INFO ][transport ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
10.1.4.220:9300]}
[2013-05-09 03:45:40,933][INFO ][cluster.service ] [Baron
Strucker] detected_master [Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]], added
{[Puishannt][9ogJ5QcyTHiWhMe-mJCPkQ][inet[/10.1.4.221:9300]],[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]],}, reason:
zen-disco-receive(from master [[Corbo,
Adrian][EtCRx4NOTwuI3bHAn8IGRg][inet[/10.1.4.222:9300]]])
[2013-05-09 03:45:41,031][INFO ][discovery ] [Baron
Strucker] _testinges.0.90.0/THfT5ZzSRPai-PV9bLaP_Q
[2013-05-09 03:45:41,042][INFO ][http ] [Baron
Strucker] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
10.1.4.220:9200]}
[2013-05-09 03:45:41,042][INFO ][node ] [Baron
Strucker] {0.90.0}[8440]: started
[2013-05-09 03:46:18,292][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [68] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@28fa1b85,
error [false], resetting
[2013-05-09 03:46:18,304][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [67] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@1441e2aa,
error [false], resetting
[2013-05-09 03:46:18,391][DEBUG][action.search.type ] [Baron
Strucker] [i_product_ys_5][2], node[EtCRx4NOTwuI3bHAn8IGRg], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@71881149]
org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize response of type [org.elasticsearch.search.dfs.DfsSearchResult]
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize response of type
[org.elasticsearch.search.dfs.DfsSearchResult]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:150)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:127)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Expected handle header, got [8]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.search.dfs.DfsSearchResult.readFrom(DfsSearchResult.java:140)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)
... 23 more
[2013-05-09 03:46:18,399][WARN ][transport.netty ] [Baron
Strucker] Message not fully read (response) for [70] handler
org.elasticsearch.search.action.SearchServiceTransportAction$1@326d234c,
error [false], resetting

I found this issue after upgrading the elasticsearch version from 0.19.10
to 0.90.0.
I have no idea why this happens in 0.90.0.
I've verified that only multiple shards will have this problem. Single
shard works fine.
Also, it happens only for my client with JAVA API.

I've tried to re-produce this issue at
Shooting Message not fully read (response) for handler org.elasticsearch.search.action.SearchServiceTransportAction · GitHub

For my client and server setting,
The client es jar version is :

org.elasticsearch
elasticsearch
0.90.0

The es servers are using 0.90.0.
ref : Elasticsearch Platform — Find real-time answers at scale | Elastic
All the java versions on the clusters are the same.

Can I ask is there any problem for my settings?

Many thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

deer · June 5, 2013, 3:22am

Hi All,

I've just found that there are some fixes for related issues in
elasticsearch 0.90.1.
DFS modes can cause undefined behaviour in 0.90 #3012

NPE when using java client with DFS_QUERY_THEN_FETCH #3008

But I can still reproduce this issue after upgrading from 0.90.0 to 0.90.1.

May I know anything I can do for this?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

deer · June 5, 2013, 5:44am

Hi All,

I've just found that there are some fixes for related issues in
elasticsearch 0.90.1.
DFS modes can cause undefined behaviour in 0.90 #3012

NPE when using java client with DFS_QUERY_THEN_FETCH #3008

But I can still reproduce this issue after upgrading from 0.90.0 to 0.90.1.

May I know anything I can do for this?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

deer · June 5, 2013, 7:59am

Hi All,

I've just found that there are some fixes for related issues in
elasticsearch 0.90.1.
DFS modes can cause undefined behaviour in 0.90 #3012

NPE when using java client with DFS_QUERY_THEN_FETCH #3008

But I can still reproduce this issue after upgrading from 0.90.0 to 0.90.1.

May I know anything I can do for this?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.