Unassigned primary and replica shards in marvel indexes

Elastic Stack Elasticsearch
1

I use ES-1.7.2. I started the cluster 3-4 days back and have not restarted the cluster at all. After couple of days, Marvel stopped working. When I investigate, I saw some of the primary and replica shards are not assigned. The logs have the following messages almost every minute -

6/ev002.log.2015-10-07:[2015-10-07 23:54:37,318][ERROR][marvel.agent.exporter ] [data-6] create failure (index:[.marvel-2015.10.08] type: [node_stats]): UnavailableShardsException[[.marvel-2015.10.08][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@262dcb91]

Here is the shard allocation:

.marvel-2015.10.08 0 r UNASSIGNED
.marvel-2015.10.08 0 p UNASSIGNED
.marvel-2015.10.07 0 p STARTED 232749 374.5mb 127.0.1.1 data-1
.marvel-2015.10.07 0 r UNASSIGNED
.marvel-2015.10.09 0 r UNASSIGNED
.marvel-2015.10.09 0 p UNASSIGNED

How do I recover from this state? and what could have gone wrong in the first place?

Thanks,
Dash.

2

Do you have enough disk space left on your node?

3

Thanks Mark, for the response. Yes, there is still 256GB of space still left on disk.

-Dash.

4

You don't happen to have allocation disabled do you?

5

Hi Mark, was going to update you about this. I have 256G of storage left,
but that is in a big drive with 2.5T storage in total. Though Elasticsearch
is taking a small fraction of that drive, it hit the 90% watermark. I
updated the cluster setting to make the high watermark to 50G, and the
shards got assigned.

I don't understand why it doesn't assign an existing shard, even when new
data is not being added there. But, changing the threshold got me over the
hump.

Again, thanks for pointing me towards disk problems.