Error in logs: Message not fully read (response) for [2298885] handler future(org.elasticsearch.indices.recovery.RecoveryTarget

Hi,

Our Elastic Search cluster has just gone yellow, and I can't work out why.
A single shard on one index on one of our four servers is showing as yellow
(INITIALIZING) and has been that way for ~20 minutes.

In the log file on that server I'm seeing:

[12:13:42,188][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289097] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@7f0608e3, error [false],resetting
[12:13:42,470][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289095] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@1455d93e, error [false],resetting
[12:13:42,667][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289096] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@45581337, error [false],resetting
[12:14:02,199][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289369] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@7e2584ea, error [false],resetting
[12:14:02,200][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289371] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@51395f59, error [false],resetting
[12:14:02,431][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2289370] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@2bb033a9, error [false],resetting
[12:49:20,084][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2298845] handler future(org.elasticsearch.indices.recovery.
RecoveryTarget$3@3bdd6de), error [true], resetting
[12:49:21,204][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2298852] handler future(org.elasticsearch.indices.recovery.
RecoveryTarget$3@4ed4f235), error [true], resetting
[12:49:23,399][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2298860] handler future(org.elasticsearch.indices.recovery.
RecoveryTarget$3@39f1c3c8), error [true], resetting
[12:49:26,557][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2298885] handler future(org.elasticsearch.indices.recovery.
RecoveryTarget$3@63afb865), error [true], resetting
[13:04:16,616][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2305703] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@23a15cb3, error [false],resetting
[13:04:16,616][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2305702] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@36ccfe0b, error [false],resetting
[13:04:23,550][WARN ][transport.netty ] [prd-elastic-2] Message notfully read
(response) for [2305704] handler org.elasticsearch.action.support.nodes.
TransportNodesOperationAction$AsyncAction$4@66fcc314, error [false],resetting

Not entirely sure what this means or how to fix it… a preliminary Google
search suggests this may be caused by mixed Elastic Search versions, but
all nodes are running 0.19.12.

Any advice would be welcomed.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You absolutely sure about the node versions all being the same? Client
nodes too?

Every time I've seen that error it has been mixed versions. Also, check
that you're using the same version of Java on all nodes.

You may be able to use the cluster reroute API to force the shard to a
different node, to see if it recovers:
http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html

And of course, a number of bugs have been fixed - look at upgrading to
0.20.5.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yeah, all using the same version. Curl'ed / on of each of them in turn and
checked the response.

In terms of Java, three of them are using 1.6.0_24, and one is
using 1.6.0_27. Could that be the cause of the problem?

The health of the cluster has since gone back to green, and we're going to
rebuild the entire cluster soon using the latest ElasticSearch and Java
versions… Hoping that will fix it!

Thanks,

Ollie

On Wednesday, March 6, 2013 6:36:25 PM UTC, Clinton Gormley wrote:

You absolutely sure about the node versions all being the same? Client
nodes too?

Every time I've seen that error it has been mixed versions. Also, check
that you're using the same version of Java on all nodes.

You may be able to use the cluster reroute API to force the shard to a
different node, to see if it recovers:
http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html

And of course, a number of bugs have been fixed - look at upgrading to
0.20.5.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Ollie

In terms of Java, three of them are using 1.6.0_24, and one is using
1.6.0_27. Could that be the cause of the problem?

I've seen differences in Java versions cause this problem as well.
Everything should be the same version, just to be safe :slight_smile:

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

And note that you should update to the latest patch (update 43 is the latest I think)

http://www.oracle.com/technetwork/java/javase/releasenotes-136954.html

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 mars 2013 à 19:30, Clinton Gormley clint@traveljury.com a écrit :

Hi Ollie

In terms of Java, three of them are using 1.6.0_24, and one is using
1.6.0_27. Could that be the cause of the problem?

I've seen differences in Java versions cause this problem as well.
Everything should be the same version, just to be safe :slight_smile:

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.