# Unassigned shards due to FailedNodeException
**URL:** https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294
**Category:** Elasticsearch
**Created:** [March 20, 2017, 4:54pm UTC](https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294 "2017-03-20T16:54:22Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [March 20, 2017, 4:54pm UTC](https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294/1 "2017-03-20T16:54:22Z")
Started a couple of simple 5.2.0 clusters of three identical nodes, where
each node is master eligible and has data. There are no restarts or
rebalancing going on. Data set is relatively small.
The indexing process for a new index is the standard of setting number of
replicas to 0 at the start and then increasing it to the proper amount when
done. The cluster has an overallocation of shards, 5, for the number of
nodes. 3. The number of replicas is set to the number of nodes minus 1,
which is 2.
Upon increasing the number of replicas, most shards will initialized and
assigned their replicas shards, except for 2 shards. Any attempts to set
the number of replicas to 0 and back to 2 will cause the same shards not to
replicate:
Nodes
ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m
node.role master name
ip.ip.ip.1 25 98 3 0.03 0.02 0.05 mdi
-
```
host1
```
ip.ip.ip.2 54 99 1 0.05 0.03 0.05 mdi
-
```
host2
```
ip.ip.ip.3 32 98 2 0.01 0.02 0.05 mdi
-
```
host3
```
Two sample indices:
health status index uuid pri rep docs.count
docs.deleted store.size pri.store.size
yellow open index1 6n2fqIziSe-vMYvrI\_HXYQ 5 2 9957799
0 24.2gb 9.3gb
yellow open index2 A\_VHoPMxRj-OnRCa4tqA9g 5 2 9957799
0 24.2gb 9.3gb
Shard status for said indicies:
index shard prirep state docs store ip node
index2 0 p STARTED 1990082 1.8gb ip.ip.ip.1 host1
index2 0 r STARTED 1990082 1.8gb ip.ip.ip.2 host2
index2 0 r STARTED 1990082 1.8gb ip.ip.ip.3 host3
index2 1 p STARTED 1990050 1.8gb ip.ip.ip.3 host3
index2 1 r STARTED 1990050 1.8gb ip.ip.ip.2 host2
index2 1 r STARTED 1990050 1.8gb ip.ip.ip.1 host1
index2 2 p STARTED 1996938 1.8gb ip.ip.ip.2 host2
index2 2 r UNASSIGNED
index2 2 r UNASSIGNED
index2 3 p STARTED 1989843 1.8gb ip.ip.ip.1 host1
index2 3 r STARTED 1989843 1.8gb ip.ip.ip.2 host2
index2 3 r STARTED 1989843 1.8gb ip.ip.ip.3 host3
index2 4 p STARTED 1990886 1.8gb ip.ip.ip.3 host3
index2 4 r STARTED 1990886 1.8gb ip.ip.ip.2 host2
index2 4 r STARTED 1990886 1.8gb ip.ip.ip.1 host1
index1 0 p STARTED 1990082 1.8gb ip.ip.ip.1 host1
index1 0 r STARTED 1990082 1.8gb ip.ip.ip.2 host2
index1 0 r STARTED 1990082 1.8gb ip.ip.ip.3 host3
index1 1 p STARTED 1990050 1.8gb ip.ip.ip.3 host3
index1 1 r STARTED 1990050 1.8gb ip.ip.ip.2 host2
index1 1 r STARTED 1990050 1.8gb ip.ip.ip.1 host1
index1 2 p STARTED 1996938 1.8gb ip.ip.ip.2 host2
index1 2 r UNASSIGNED
index1 2 r UNASSIGNED
index1 3 p STARTED 1989843 1.8gb ip.ip.ip.1 host1
index1 3 r STARTED 1989843 1.8gb ip.ip.ip.2 host2
index1 3 r STARTED 1989843 1.8gb ip.ip.ip.3 host3
index1 4 p STARTED 1990886 1.8gb ip.ip.ip.3 host3
index1 4 r STARTED 1990886 1.8gb ip.ip.ip.2 host2
index1 4 r STARTED 1990886 1.8gb ip.ip.ip.1 host1
The most relevant stack trace in the logs are:
Caused by: org.elasticsearch.ElasticsearchException: Failed to list store
metadata for shard [[index1][1]]
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:114)
~[elasticsearch-5.2.0.jar:5.2.0]
...
Caused by: java.io.FileNotFoundException: no segments\* file found in
store(mmapfs(/elasticsearchdata/nodes/0/indices/6n2fqIziSe-vMYvrI\_HXYQ/1/index)):
files: [recovery.AVrepWc\_SlfGaVklv4AX.\_0.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_0.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_0.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_1.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_1.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_2.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_2.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_2.si, recovery.AVrepWc\_SlfGaVklv4AX.\_3.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_3.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_3.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_4.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_4.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_4.si, recovery.AVrepWc\_SlfGaVklv4AX.\_5.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_5.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_5.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_6.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_6.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_6.si, recovery.AVrepWc\_SlfGaVklv4AX.\_7.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_7.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_7.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_8.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_8.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_8.si, recovery.AVrepWc\_SlfGaVklv4AX.\_9.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_9.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_9.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_a.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_a.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_a.si, recovery.AVrepWc\_SlfGaVklv4AX.\_b.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_b.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_b.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_c.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_c.si,
recovery.AVrepWc\_SlfGaVklv4AX.\_d.cfe, recovery.AVrepWc\_SlfGaVklv4AX.\_d.cfs,
recovery.AVrepWc\_SlfGaVklv4AX.\_d.si, recovery.AVrepWc\_SlfGaVklv4AX.\_e.cfe,
recovery.AVrepWc\_SlfGaVklv4AX.\_e.cfs, recovery.AVrepWc\_SlfGaVklv4AX.\_e.si,
recovery.AVrepWc\_SlfGaVklv4AX.segments\_4, write.lock]
Larger stack trace and the above outputs:
>
>
> There are more than three files. show original
This behavior only occurs on one of the clusters consistently for each
index, while the other cluster works as expected. Both are provisioned
identically.
Cheers,
Ivan
---
### Author:  [@ywelsch](https://discuss.elastic.co/u/ywelsch)
#### Post date: [March 24, 2017, 9:16am UTC](https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294/2 "2017-03-24T09:16:23Z")
Closed in [https://github.com/elastic/elasticsearch/issues/23676](https://github.com/elastic/elasticsearch/issues/23676)
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [March 24, 2017, 4:34pm UTC](https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294/3 "2017-03-24T16:34:21Z")
Yannick, thanks for helping and closing. I was not able to comment since
the mailing list does not send out an email to the sender when posted, so I
had nothing to reply to.
Ivan
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 21, 2017, 4:34pm UTC](https://discuss.elastic.co/t/unassigned-shards-due-to-failednodeexception/79294/4 "2017-04-21T16:34:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.