Shard Allocation Problem

Marcelo_Paes_Rech · March 28, 2015, 7:29pm

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I am
including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

gist.github.com

https://gist.github.com/marcelopaesrech/7b9f94310ff7a24bed9e

gistfile1.txt

[2015-03-24 15:26:05,244][DEBUG][indices.cluster          ] [THE_CHOSEN_ONE] [my_index][0] creating shard
[2015-03-24 15:26:05,244][DEBUG][index.service            ] [THE_CHOSEN_ONE] [my_index] creating shard_id [0]
[2015-03-24 15:26:05,274][DEBUG][index.deletionpolicy     ] [THE_CHOSEN_ONE] [my_index][0] Using [keep_only_last] deletion policy
[2015-03-24 15:26:05,275][DEBUG][index.merge.policy       ] [THE_CHOSEN_ONE] [my_index][0] using [tiered] merge policy with expunge_deletes_allowed[10.0], floor_segment[2mb], max_merge_at_once[10], max_merge_at_once_explicit[30], max_merged_segment[5gb], segments_per_tier[10.0], reclaim_deletes_weight[2.0]
[2015-03-24 15:26:05,276][DEBUG][index.merge.scheduler    ] [THE_CHOSEN_ONE] [my_index][0] using [concurrent] merge scheduler with max_thread_count[3]
[2015-03-24 15:26:05,278][DEBUG][index.shard.service      ] [THE_CHOSEN_ONE] [my_index][0] state: [CREATED]
[2015-03-24 15:26:05,278][DEBUG][index.translog           ] [THE_CHOSEN_ONE] [my_index][0] interval [5s], flush_threshold_ops [5000], flush_threshold_size [200mb], flush_threshold_period [30m]
[2015-03-24 15:26:05,280][DEBUG][index.shard.service      ] [THE_CHOSEN_ONE] [my_index][0] state: [CREATED]->[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
[2015-03-24 15:26:05,281][DEBUG][indices.cluster          ] [THE_CHOSEN_ONE] [my_index][2] creating shard
[2015-03-24 15:26:05,281][DEBUG][index.service            ] [THE_CHOSEN_ONE] [my_index] creating shard_id [2]

This file has been truncated. show original

Master Node

gist.github.com

https://gist.github.com/marcelopaesrech/c80542a85c83d9c1756d

LOG Master Node 10281

[2015-03-26 15:23:08,468][WARN ][cluster.action.shard     ] [MR3] [my_index][1] received shard failed for [my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,506][WARN ][cluster.action.shard     ] [MR3] [my_index][1] received shard failed for [my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,516][WARN ][cluster.action.shard     ] [MR3] [my_index][1] received shard failed for [my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,523][WARN ][cluster.action.shard     ] [MR3] [my_index][1] received shard failed for [my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,534][WARN ][cluster.action.shard     ] [MR3] [my_index][1] received shard failed for [my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,545][DEBUG][cluster.service          ] [MR3] set local cluster state to version 142212
[2015-03-26 15:23:08,545][DEBUG][river.cluster            ] [MR3] processing [reroute_rivers_node_changed]: execute
[2015-03-26 15:23:08,545][DEBUG][river.cluster            ] [MR3] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-03-26 15:23:08,547][DEBUG][cluster.service          ] [MR3] processing [shard-started ([my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], [R], s[INITIALIZING]), reason [after recovery (replica) from node [[KOZANI][_dLxnGwlTt6P4i7PqzNEUQ][pves1-6.popvono][inet[/172.31.234.187:9300]]{master=false}]]]: done applying updated cluster_state (version: 142212)
[2015-03-26 15:23:08,547][DEBUG][cluster.service          ] [MR3] processing [shard-failed ([my_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute

This file has been truncated. show original

Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e15b7401-259b-409d-8631-63642d2b695e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcelo_Paes_Rech · April 1, 2015, 6:45pm

The Elasticsearch Team closed the issue, but I didn't have an answer. I
still looking for a solution.

Regards.

Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I
am including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

Data node 10281 · GitHub

Master Node

LOG Master Node 10281 · GitHub

Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · April 1, 2015, 11:21pm

It looks like this is only happening for replicas. I'd suggest picking a
single index, remove the replicas and then re-add them and see if they are
allocated successfully.

On 2 April 2015 at 05:45, Marcelo Paes Rech marcelopaesrech@gmail.com
wrote:

The Elasticsearch Team closed the issue, but I didn't have an answer. I
still looking for a solution.

Regards.

Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I
am including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

Data node 10281 · GitHub

Master Node

LOG Master Node 10281 · GitHub

Regards.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-RicbxJ_azuKDPPjTX11tX2baXYRQpJqNBvgHTPtC1bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.