Shards UNASSIGNED even tho they exist on disk

Hi,

I have a problem with a few index, some of the shards (both replica and
primary) are UNASSIGNED, my cluster stays yellow.

This is what the master says about that:
[2014-11-10 06:53:01,223][WARN ][cluster.action.shard ] [node-master]
[index][9] received shard failed for [index][9],
node[9g2_kOrDSt-57UVI1bLfFg], [P], s[STARTED], indexUUID
[20P5SMNFTZyrUEVyUPCsbQ], reason [master
[node-master][07ZcjsurR3iIVsH6iSX0jw][data-node][inet[/xx.xx.xx.xx:9300]]{data=false,
master=true} marked shard as started, but shard has not been created, mark
shard as failed]

http://host:9200/index/_stats"_shards":{"failed":0,"successful":13,"total":
20

This happend when i dropped a node, and let it replicate itself together,
replication factor is 1 (two shards identical)
I did it on two nodes, worked perfectly, then on the third node, i have 92
SHARDS Unassigned

The only different between the first two nodes and the third is that it ran
with these settings:

          "cluster.routing.allocation.disk.threshold_enabled": true,

      "cluster.routing.allocation.disk.watermark.low": "0.85",

    "cluster.routing.allocation.disk.watermark.high": "0.90",

    "cluster.info.update.interval": "60s",

"indices.recovery.concurrent_streams": "10",

"cluster.routing.allocation.node_concurrent_recoveries": "40",

Any idea if this can be fixed?

Ive tried to clean up the masters and restarted them, nothing
Ive tried to delete _state on data-node on these index, nothing

Thanks for help :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1706f0bc-5aef-42dc-bcbe-8f62efd25faf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Another thing i just noticed..

The first problematic index with the problem, is from 2014-10-08,
We upgraded to 1.3.2 from 1.01 2014-10-07, indexes is created nighthly.

Den måndagen den 10:e november 2014 kl. 14:45:09 UTC+1 skrev Johan Öhr:

Hi,

I have a problem with a few index, some of the shards (both replica and
primary) are UNASSIGNED, my cluster stays yellow.

This is what the master says about that:
[2014-11-10 06:53:01,223][WARN ][cluster.action.shard ] [node-master]
[index][9] received shard failed for [index][9],
node[9g2_kOrDSt-57UVI1bLfFg], [P], s[STARTED], indexUUID
[20P5SMNFTZyrUEVyUPCsbQ], reason [master
[node-master][07ZcjsurR3iIVsH6iSX0jw][data-node][inet[/xx.xx.xx.xx:9300]]{data=false,
master=true} marked shard as started, but shard has not been created, mark
shard as failed]

http://host:9200/index/_stats"_shards":{"failed":0,"successful":13,
"total":20

This happend when i dropped a node, and let it replicate itself together,
replication factor is 1 (two shards identical)
I did it on two nodes, worked perfectly, then on the third node, i have 92
SHARDS Unassigned

The only different between the first two nodes and the third is that it
ran with these settings:

          "cluster.routing.allocation.disk.threshold_enabled": true,

      "cluster.routing.allocation.disk.watermark.low": "0.85",

    "cluster.routing.allocation.disk.watermark.high": "0.90",

    "cluster.info.update.interval": "60s",

"indices.recovery.concurrent_streams": "10",

"cluster.routing.allocation.node_concurrent_recoveries": "40",

Any idea if this can be fixed?

Ive tried to clean up the masters and restarted them, nothing
Ive tried to delete _state on data-node on these index, nothing

Thanks for help :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03686f7e-df0b-4fcd-89fd-ebc66a0ebc05%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sounds like you don't have enough space on disk(s)

Happened to me when upgrading to 1.4.

On Monday, November 10, 2014 2:45:09 PM UTC+1, Johan Öhr wrote:

Hi,

I have a problem with a few index, some of the shards (both replica and
primary) are UNASSIGNED, my cluster stays yellow.

This is what the master says about that:
[2014-11-10 06:53:01,223][WARN ][cluster.action.shard ] [node-master]
[index][9] received shard failed for [index][9],
node[9g2_kOrDSt-57UVI1bLfFg], [P], s[STARTED], indexUUID
[20P5SMNFTZyrUEVyUPCsbQ], reason [master
[node-master][07ZcjsurR3iIVsH6iSX0jw][data-node][inet[/xx.xx.xx.xx:9300]]{data=false,
master=true} marked shard as started, but shard has not been created, mark
shard as failed]

http://host:9200/index/_stats"_shards":{"failed":0,"successful":13,
"total":20

This happend when i dropped a node, and let it replicate itself together,
replication factor is 1 (two shards identical)
I did it on two nodes, worked perfectly, then on the third node, i have 92
SHARDS Unassigned

The only different between the first two nodes and the third is that it
ran with these settings:

          "cluster.routing.allocation.disk.threshold_enabled": true,

      "cluster.routing.allocation.disk.watermark.low": "0.85",

    "cluster.routing.allocation.disk.watermark.high": "0.90",

    "cluster.info.update.interval": "60s",

"indices.recovery.concurrent_streams": "10",

"cluster.routing.allocation.node_concurrent_recoveries": "40",

Any idea if this can be fixed?

Ive tried to clean up the masters and restarted them, nothing
Ive tried to delete _state on data-node on these index, nothing

Thanks for help :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c1c3bd8-1e94-471d-ab06-fc643afe836e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.