Sending failed shard error

Elastic Stack Elasticsearch
1

Hi i have installed the new Elasticsearch 1.0. I am trying to setup cluster
with multicast disabled and unicast ip set. i was successfully able to
index data when only one node is started. But when i start the second i am
getting the following error

sending failed shard for [twitter][2], node[HCvvTxyASOmIzlljKQn_og], [R],
s[INITIALIZING], indexUUID [jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to
start shard, message [RecoveryFailedException[[twitter][2]: Recovery failed
from [7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

in the second

[twitter][2] received shard failed for [twitter][2],
node[HCvvTxyASOmIzlljKQn_og], [R], s[INITIALIZING], indexUUID
[jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to start shard, message
[RecoveryFailedException[[twitter][2]: Recovery failed from
[7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

Tried cleaning the data folder and restarting the cluster. but still facing
the same problem. Can anyone explain this and a solution for the same

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

It sounds like you have multiple versions of Elasticsearch running with the
same cluster name.

clint

On 5 March 2014 12:29, Hari Prasad iamhari1234@gmail.com wrote:

Hi i have installed the new Elasticsearch 1.0. I am trying to setup
cluster with multicast disabled and unicast ip set. i was successfully able
to index data when only one node is started. But when i start the second i
am getting the following error

sending failed shard for [twitter][2], node[HCvvTxyASOmIzlljKQn_og], [R],
s[INITIALIZING], indexUUID [jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to
start shard, message [RecoveryFailedException[[twitter][2]: Recovery failed
from [7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

in the second

[twitter][2] received shard failed for [twitter][2],
node[HCvvTxyASOmIzlljKQn_og], [R], s[INITIALIZING], indexUUID
[jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to start shard, message
[RecoveryFailedException[[twitter][2]: Recovery failed from
[7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

Tried cleaning the data folder and restarting the cluster. but still
facing the same problem. Can anyone explain this and a solution for the same

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKS%2B1ORhxCdxLnf_fFYL-Pn5eicrWuYWWtht%2BQh7zsFGLw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

But the different ES have different clusters. will that still cause this
problem?

On Wednesday, 5 March 2014 17:02:27 UTC+5:30, Clinton Gormley wrote:

It sounds like you have multiple versions of Elasticsearch running with
the same cluster name.

clint

On 5 March 2014 12:29, Hari Prasad <iamha...@gmail.com <javascript:>>wrote:

Hi i have installed the new Elasticsearch 1.0. I am trying to setup
cluster with multicast disabled and unicast ip set. i was successfully able
to index data when only one node is started. But when i start the second i
am getting the following error

sending failed shard for [twitter][2], node[HCvvTxyASOmIzlljKQn_og], [R],
s[INITIALIZING], indexUUID [jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to
start shard, message [RecoveryFailedException[[twitter][2]: Recovery failed
from [7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

in the second

[twitter][2] received shard failed for [twitter][2],
node[HCvvTxyASOmIzlljKQn_og], [R], s[INITIALIZING], indexUUID
[jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to start shard, message
[RecoveryFailedException[[twitter][2]: Recovery failed from
[7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

Tried cleaning the data folder and restarting the cluster. but still
facing the same problem. Can anyone explain this and a solution for the same

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e55633a-d392-4dd0-8ca7-a9f65918c2d9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

If they have the same cluster name, they will try to join each other.

On 5 March 2014 12:36, Hari Prasad iamhari1234@gmail.com wrote:

But the different ES have different clusters. will that still cause this
problem?

On Wednesday, 5 March 2014 17:02:27 UTC+5:30, Clinton Gormley wrote:

It sounds like you have multiple versions of Elasticsearch running with
the same cluster name.

clint

On 5 March 2014 12:29, Hari Prasad iamha...@gmail.com wrote:

Hi i have installed the new Elasticsearch 1.0. I am trying to setup
cluster with multicast disabled and unicast ip set. i was successfully able
to index data when only one node is started. But when i start the second i
am getting the following error

sending failed shard for [twitter][2], node[HCvvTxyASOmIzlljKQn_og],
[R], s[INITIALIZING], indexUUID [jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to
start shard, message [RecoveryFailedException[[twitter][2]: Recovery
failed from [7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

in the second

[twitter][2] received shard failed for [twitter][2],
node[HCvvTxyASOmIzlljKQn_og], [R], s[INITIALIZING], indexUUID
[jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to start shard, message
[RecoveryFailedException[[twitter][2]: Recovery failed from
[7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

Tried cleaning the data folder and restarting the cluster. but still
facing the same problem. Can anyone explain this and a solution for the same

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4e55633a-d392-4dd0-8ca7-a9f65918c2d9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/4e55633a-d392-4dd0-8ca7-a9f65918c2d9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKRJZZ8eV%3D3dXJyDSbVCEE6CNqqj-4QsukObeXveTYsMdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

My clusters have different names, also each of their multicast is false.

On Wednesday, 5 March 2014 17:06:47 UTC+5:30, Clinton Gormley wrote:

If they have the same cluster name, they will try to join each other.

On 5 March 2014 12:36, Hari Prasad <iamha...@gmail.com <javascript:>>wrote:

But the different ES have different clusters. will that still cause this
problem?

On Wednesday, 5 March 2014 17:02:27 UTC+5:30, Clinton Gormley wrote:

It sounds like you have multiple versions of Elasticsearch running with
the same cluster name.

clint

On 5 March 2014 12:29, Hari Prasad iamha...@gmail.com wrote:

Hi i have installed the new Elasticsearch 1.0. I am trying to setup
cluster with multicast disabled and unicast ip set. i was successfully able
to index data when only one node is started. But when i start the second i
am getting the following error

sending failed shard for [twitter][2], node[HCvvTxyASOmIzlljKQn_og],
[R], s[INITIALIZING], indexUUID [jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to
start shard, message [RecoveryFailedException[[twitter][2]: Recovery
failed from [7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

in the second

[twitter][2] received shard failed for [twitter][2],
node[HCvvTxyASOmIzlljKQn_og], [R], s[INITIALIZING], indexUUID
[jqLwCz5QSjCo9P1nAmoT0g], reason [Failed to start shard, message
[RecoveryFailedException[[twitter][2]: Recovery failed from
[7vfQ40p2SfKRtLOg3DQBbQ][localhost.localdomain] into
[HCvvTxyASOmIzlljKQn_og][localhost.localdomain]; nested:
RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[Failed to deserialize
exception response from stream]; nested: EOFException; ]]

Tried cleaning the data folder and restarting the cluster. but still
facing the same problem. Can anyone explain this and a solution for the same

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/28d4a402-45c9-4c37-8fe8-d46073e7492e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4e55633a-d392-4dd0-8ca7-a9f65918c2d9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/4e55633a-d392-4dd0-8ca7-a9f65918c2d9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d8d3930-7855-489f-bd21-c0ceb8e90b3c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

On 5 March 2014 12:47, Hari Prasad iamhari1234@gmail.com wrote:

My clusters have different names, also each of their multicast is false.

There is another node somewhere that is interfering. Have a look in the
logs, or the output from:

curl 'localhost:9200/_nodes?all&pretty'

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKTKE9vSdimoQ6vUnqVH3mV3zAa_UhJ3P7L--RszMiKtGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

I am still getting that error. The output of the command listed only those
nodes that should be in my cluster. I even checked the version of each
node. so i believe no other node.
But finally when i view my cluster through head plugin everything is fine.

On Wednesday, 5 March 2014 17:21:04 UTC+5:30, Clinton Gormley wrote:

On 5 March 2014 12:47, Hari Prasad <iamha...@gmail.com <javascript:>>wrote:

My clusters have different names, also each of their multicast is false.

There is another node somewhere that is interfering. Have a look in the
logs, or the output from:

curl 'localhost:9200/_nodes?all&pretty'

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23e59d31-ad2f-421b-812b-d9a6840070e8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

I'd also check the Java version and make sure they are the same on all
nodes.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55d38bfb-2a52-4c00-bb87-3a5ff662812e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

The java in all the nodes are 1.7

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/354fd9fa-43a4-4650-92f3-78e74bd33a1b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

I even tried a killall -9 java. then started only the elasticsearch 1.0
processes. the nodes joined the cluster.
In head plugin i am able to see only the node that should be there.
when i perform the insert few shards are getting replicated but for few i
am again getting the same error.

On Wednesday, 5 March 2014 20:06:30 UTC+5:30, Hari Prasad wrote:

The java in all the nodes are 1.7

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/891ce62e-d60f-4e95-be4e-764f0bc6e11d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Can you also confirm that the Java 1.7 update versions are all the same,
i.e. Java 1.7.0u_XXX?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7da7d28-5bb6-4449-9e99-f73ee017fd65%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Thanks Binh
It seems thats the problem. in one machine the version was 1.7.0 and in
another one the version was 1.7.0_45.
If you don't mind can you please explain the root cause, as in what
differed in the 2 java version

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f636c3f-d107-4d06-81de-60b4172014ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

13

I don't know specifically what changed from 0 to 45 but as the JVM bugs are
fixed from update to update, it is likely that some kind of low-level
object serialization code will change, which is what your error message
seems to indicate.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/787c5c3b-a584-4821-a250-96fdbed6a58d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

14

Thank you Binh Ly

On Friday, 7 March 2014 19:12:29 UTC+5:30, Binh Ly wrote:

I don't know specifically what changed from 0 to 45 but as the JVM bugs
are fixed from update to update, it is likely that some kind of low-level
object serialization code will change, which is what your error message
seems to indicate.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46bc25bf-81bd-4740-a208-9397e618e139%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.