Shards get stuck in RELOCATING state

Elastic Stack Elasticsearch
1

Hi,
I see two shards are stuck in RELOCATING state, for more than 18 hours as of now.
curl localhost:9200/_status shows something like this:
"7" : [ {
"routing" : {
"state" : "RELOCATING",
"primary" : false,
"node" : "fyNmNb0yRnuVq-xHLFNU-Q",
"relocating_node" : "Gm7QcAG0Tom-kZFNfL-IGQ",
"shard" : 7,
"index" : "i42"
},

and it does not change.

I don't see any errors in logs.
(indices.recovery log level is default)

indexing and search seem working, but not sure for the shards under relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" : 10}}'
does not come back. (does not even timeout)

What is the best way to make it proceed?

Thanks for your help.

2

Quick question: do you have enough disk space on all the nodes?

--
Ivan

On Tue, Nov 20, 2012 at 10:35 AM, arta artasano@sbcglobal.net wrote:

Hi,
I see two shards are stuck in RELOCATING state, for more than 18 hours as
of
now.
curl localhost:9200/_status shows something like this:
"7" : [ {
"routing" : {
"state" : "RELOCATING",
"primary" : false,
"node" : "fyNmNb0yRnuVq-xHLFNU-Q",
"relocating_node" : "Gm7QcAG0Tom-kZFNfL-IGQ",
"shard" : 7,
"index" : "i42"
},

and it does not change.

I don't see any errors in logs.
(indices.recovery log level is default)

What is the best way to make it proceed?

Thanks for your help.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Shards-get-stuck-in-RELOCATING-state-tp4025778.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--

--

3

Thanks Ivan. Yes I have enough diskspace.

4

Additional notes:

indexing and search seem working, but not sure for the shards under relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" : 10}}'
does not come back. (does not even timeout)

5

Hi,

Are you able to give us some information about what version of
Elasticsearch you're using and any information about your cluster layout /
network layout.

On Wednesday, November 21, 2012 11:02:05 AM UTC+13, arta wrote:

Additional notes:

indexing and search seem working, but not sure for the shards under
relocation.
curl command such as
curl -XPUT localhost:9200/_cluster/settings -d
'{"transient":{"cluster.routing.allocation.cluster_concurrent_rebalance" :
10}}'
does not come back. (does not even timeout)

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Shards-get-stuck-in-RELOCATING-state-tp4025778p4025783.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--

6

Hi Chris, thanks for the reply.

I'm using 0.19.3.
I have 19 data nodes in one cluster, all are master eligible, they are in our private network.
Is this enough info for you?

7

Hi!

Am Donnerstag, 22. November 2012 01:14:37 UTC+1 schrieb arta:

Hi Chris, thanks for the reply.

I'm using 0.19.3.
I have 19 data nodes in one cluster, all are master eligible, they are in
our private network.
Is this enough info for you?

We are running an 8 node cluster and I must say that it was not very stable
before we installed 0.19.11 (we jumped from 0.19.4). Now we are running
0.19.11 and it is rock solid. We had similar problems with stuck shards.

CU
Thomas

--