Shard Allocation Problem

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I am
including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

Master Node

Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e15b7401-259b-409d-8631-63642d2b695e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The ElasticSearch Team closed the issue, but I didn't have an answer. I
still looking for a solution.

Regards.

Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I
am including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

https://gist.github.com/marcelopaesrech/7b9f94310ff7a24bed9e

Master Node

https://gist.github.com/marcelopaesrech/c80542a85c83d9c1756d

Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It looks like this is only happening for replicas. I'd suggest picking a
single index, remove the replicas and then re-add them and see if they are
allocated successfully.

On 2 April 2015 at 05:45, Marcelo Paes Rech marcelopaesrech@gmail.com
wrote:

The ElasticSearch Team closed the issue, but I didn't have an answer. I
still looking for a solution.

Regards.

Em sábado, 28 de março de 2015 16:29:05 UTC-3, Marcelo Paes Rech escreveu:

Hi guys,

Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it never finishes rebalancing.

Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.

I have created #10281
https://github.com/elastic/elasticsearch/issues/10281 on GitHub, but I
am including the problem here too. Maybe somebody ran into this problem and
already fixed it.

The logs were collected in different moments but they represent the same
behaviour.

The log follows:

Data Node

https://gist.github.com/marcelopaesrech/7b9f94310ff7a24bed9e

Master Node

https://gist.github.com/marcelopaesrech/c80542a85c83d9c1756d

Regards.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d7e23198-fd2a-4b41-aeed-cf9b8cc23ae7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-RicbxJ_azuKDPPjTX11tX2baXYRQpJqNBvgHTPtC1bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.