# Could not reassign UNASSIGNED shards (elasticsearch 5.6 )
**URL:** https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154
**Category:** Elasticsearch
**Created:** [December 25, 2017, 3:08pm UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154 "2017-12-25T15:08:40Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 25, 2017, 3:08pm UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/1 "2017-12-25T15:08:41Z")
After reboot, i got some unassigned shards.
> logstash-2017.12.08 3 p UNASSIGNED
> logstash-2017.12.08 3 r UNASSIGNED
> logstash-2017.12.08 4 p UNASSIGNED
> logstash-2017.12.08 4 r UNASSIGNED
> logstash-2017.12.08 2 p UNASSIGNED
> logstash-2017.12.08 2 r UNASSIGNED
> logstash-2017.12.08 1 p UNASSIGNED
> logstash-2017.12.08 1 r UNASSIGNED
> logstash-2017.12.08 0 p UNASSIGNED
> logstash-2017.12.08 0 r UNASSIGNED
now i am trying to assigned it but i got
> root@04elasticsearch:~# curl -XPOST '[http://localhost:9200/\_cluster/reroute](http://localhost:9200/_cluster/reroute)' -d '{ "commands" : [{ "index" : "logstash-2017.12.08", "allocate" : { "shard" : 3, "node" : "7vuMfymHTTOzxlcTtAkk9g", "allow\_primary" : true } }]}' | jq .
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 514 100 368 100 146 62542 24813 --:--:-- --:--:-- --:--:-- 73600
> {
> "status": 400,
> "error": {
> "caused\_by": {
> "col": 31,
> "line": 1,
> "reason": "Unknown AllocationCommand [index]",
> "type": "unknown\_named\_object\_exception"
> },
> "col": 31,
> "line": 1,
> "reason": "[cluster\_reroute] failed to parse field [commands]",
> "type": "parsing\_exception",
> "root\_cause": [
> {
> "col": 31,
> "line": 1,
> "reason": "Unknown AllocationCommand [index]",
> "type": "unknown\_named\_object\_exception"
> }
> ]
> }
> }
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 25, 2017, 8:08pm UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/2 "2017-12-25T20:08:23Z")
After i tried
> GET /\_cluster/allocation/explain
> {
> "index": "logstash-2017.12.05",
> "shard": 0,
> "primary": true
> }
> "index": "logstash-2017.12.05",
> "shard": 0,
> "primary": true,
> "current\_state": "unassigned",
> "unassigned\_info": {
> "reason": "ALLOCATION\_FAILED",
> "at": "2017-12-25T18:48:27.248Z",
> "failed\_allocation\_attempts": 1,
> "details": "failed recovery, failure RecoveryFailedException[[logstash-2017.12.05][0]: Recovery failed on {7vuMfym}{7vuMfymHTTOzxlcTtAkk9g}{e4mS1ZIKS5aJHUA3PHTx3g}{10.3.2.45}{10.3.2.45:9300}{ml.max\_open\_jobs=10, ml.enabled=true}]; nested: IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files:]; nested: FileNotFoundException[no segments\* file found in store(mmapfs(/vol/nodes/0/indices/AgVIeZ2cQji5SSFlnmS8dw/0/index)): files:]; ",
> "last\_allocation\_status": "no\_valid\_shard\_copy"
> },
> "can\_allocate": "no\_valid\_shard\_copy",
> "allocate\_explanation": "cannot allocate because all found copies of the shard are either stale or corrupt",
> "node\_allocation\_decisions": [
> {
> "node\_id": "7vuMfymHTTOzxlcTtAkk9g",
> "node\_name": "7vuMfym",
> "transport\_address": "10.3.2.45:9300",
> "node\_attributes": {
> "ml.max\_open\_jobs": "10",
> "ml.enabled": "true"
> },
> "node\_decision": "no",
> "store": {
> "in\_sync": true,
> "allocation\_id": "Wx2RRHWjQUWO38H6u8eXnA",
> "store\_exception": {
> "type": "file\_not\_found\_exception",
> "reason": "no segments\* file found in SimpleFSDirectory@/vol/nodes/0/indices/AgVIeZ2cQji5SSFlnmS8dw/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@4ea20096: files: "
> }
> }
> },
> {
> "node\_id": "dEcFJ6h\_QfyKNr\_N7362QQ",
> "node\_name": "dEcFJ6h",
> "transport\_address": "10.3.2.39:9300",
> "node\_attributes": {
> "ml.max\_open\_jobs": "10",
> "ml.enabled": "true"
> },
> "node\_decision": "no",
> "store": {
> "found": false
> }
> }
> ]
> }
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 25, 2017, 8:10pm UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/3 "2017-12-25T20:10:43Z")
Also i tried
> POST /logstash-2017.12.05/\_close
> POST /logstash-2017.12.05/\_open
and
> PUT /logstash-2017.12.05/\_settings
> {
> "index" : {
> "number\_of\_replicas" : 0
> }
> }
'+'
> PUT /logstash-2017.12.05/\_settings
> {
> "index" : {
> "number\_of\_replicas" : 1
> }
> }
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 25, 2017, 8:12pm UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/4 "2017-12-25T20:12:30Z")
and
> POST /\_cluster/reroute
> {
> "commands" : [
> {
> "move" : {
> "index" : "logstash-2017.12.05", "shard" : 0,
> "from\_node" : "dEcFJ6h\_QfyKNr\_N7362QQ", "to\_node" : "dEcFJ6h\_QfyKNr\_N7362QQ"
> }
> },
> {
> "allocate\_replica" : {
> "index" : "logstash-2017.12.05", "shard" : 1,
> "node" : "dEcFJ6h\_QfyKNr\_N7362QQ"
> }
> }
> ]
> }
but in this case i don't know what should i put in "from node" filed i this index not assign to any server
By the way allocation for all indexes are enabled
---
### Author:  [@zqc0512](https://discuss.elastic.co/u/zqc0512)
#### Post date: [December 26, 2017, 12:44am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/5 "2017-12-26T00:44:52Z")
this you can slove it as i know.
**i think you cluster have many shards.**
i test is in my cluster find the slove it fast is close it and open again.
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 7:44am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/6 "2017-12-26T07:44:59Z")
I dont understand , explain please how to fix it
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 7:49am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/7 "2017-12-26T07:49:50Z")
What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)?
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 8:36am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/8 "2017-12-26T08:36:23Z")
> {
> "\_nodes": {
> "total": 2,
> "successful": 2,
> "failed": 0
> },
> "cluster\_name": "prod",
> "timestamp": 1514277344003,
> "status": "red",
> "indices": {
> "count": 81,
> "shards": {
> "total": 409,
> "primaries": 205,
> "replication": 0.9951219512195122,
> "index": {
> "shards": {
> "min": 1,
> "max": 10,
> "avg": 5.049382716049383
> },
> "primaries": {
> "min": 1,
> "max": 5,
> "avg": 2.5308641975308643
> },
> "replication": {
> "min": 0,
> "max": 1,
> "avg": 0.9876543209876543
> }
> }
> },
> "docs": {
> "count": 445304538,
> "deleted": 26909
> },
> "store": {
> "size": "712.4gb",
> "size\_in\_bytes": 765019382231,
> "throttle\_time": "0s",
> "throttle\_time\_in\_millis": 0
> },
> "fielddata": {
> "memory\_size": "12.7kb",
> "memory\_size\_in\_bytes": 13104,
> "evictions": 0
> },
> "query\_cache": {
> "memory\_size": "0b",
> "memory\_size\_in\_bytes": 0,
> "total\_count": 0,
> "hit\_count": 0,
> "miss\_count": 0,
> "cache\_size": 0,
> "cache\_count": 0,
> "evictions": 0
> },
> "completion": {
> "size": "0b",
> "size\_in\_bytes": 0
> },
> "segments": {
> "count": 8011,
> "memory": "1.8gb",
> "memory\_in\_bytes": 2034123647,
> "terms\_memory": "1.6gb",
> "terms\_memory\_in\_bytes": 1733836573,
> "stored\_fields\_memory": "188.9mb",
> "stored\_fields\_memory\_in\_bytes": 198105344,
> "term\_vectors\_memory": "0b",
> "term\_vectors\_memory\_in\_bytes": 0,
> "norms\_memory": "295.4kb",
> "norms\_memory\_in\_bytes": 302528,
> "points\_memory": "15.9mb",
> "points\_memory\_in\_bytes": 16733358,
> "doc\_values\_memory": "81.2mb",
> "doc\_values\_memory\_in\_bytes": 85145844,
> "index\_writer\_memory": "46.1mb",
> "index\_writer\_memory\_in\_bytes": 48391088,
> "version\_map\_memory": "81.9kb",
> "version\_map\_memory\_in\_bytes": 83886,
> "fixed\_bit\_set": "72kb",
> "fixed\_bit\_set\_memory\_in\_bytes": 73728,
> "max\_unsafe\_auto\_id\_timestamp": 1514246409906,
> "file\_sizes": {}
> }
> },
> "nodes": {
> "count": {
> "total": 2,
> "data": 2,
> "coordinating\_only": 0,
> "master": 2,
> "ingest": 2
> },
> "versions": [
> "5.5.2"
> ],
> "os": {
> "available\_processors": 16,
> "allocated\_processors": 16,
> "names": [
> {
> "name": "Linux",
> "count": 2
> }
> ],
> "mem": {
> "total": "119.9gb",
> "total\_in\_bytes": 128781852672,
> "free": "773.6mb",
> "free\_in\_bytes": 811261952,
> "used": "119.1gb",
> "used\_in\_bytes": 127970590720,
> "free\_percent": 1,
> "used\_percent": 99
> }
> },
> "process": {
> "cpu": {
> "percent": 5
> },
> "open\_file\_descriptors": {
> "min": 740,
> "max": 758,
> "avg": 749
> }
> },
> "jvm": {
> "max\_uptime": "15.6h",
> "max\_uptime\_in\_millis": 56253923,
> "versions": [
> {
> "version": "1.8.0\_144",
> "vm\_name": "Java HotSpot(TM) 64-Bit Server VM",
> "vm\_version": "25.144-b01",
> "vm\_vendor": "Oracle Corporation",
> "count": 2
> }
> ],
> "mem": {
> "heap\_used": "25.5gb",
> "heap\_used\_in\_bytes": 27453966464,
> "heap\_max": "63.8gb",
> "heap\_max\_in\_bytes": 68580016128
> },
> "threads": 230
> },
> "fs": {
> "total": "3.4tb",
> "total\_in\_bytes": 3740103417856,
> "free": "2.7tb",
> "free\_in\_bytes": 2969643327488,
> "available": "2.5tb",
> "available\_in\_bytes": 2779609776128
> },
> "plugins": [
> {
> "name": "ingest-geoip",
> "version": "5.5.2",
> "description": "Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database",
> "classname": "org.elasticsearch.ingest.geoip.IngestGeoIpPlugin",
> "has\_native\_controller": false
> },
> {
> "name": "repository-s3",
> "version": "5.5.2",
> "description": "The S3 repository plugin adds S3 repositories",
> "classname": "org.elasticsearch.repositories.s3.S3RepositoryPlugin",
> "has\_native\_controller": false
> },
> {
> "name": "x-pack",
> "version": "5.5.2",
> "description": "Elasticsearch Expanded Pack Plugin",
> "classname": "org.elasticsearch.xpack.XPackPlugin",
> "has\_native\_controller": true
> }
> ],
> "network\_types": {
> "transport\_types": {
> "security4": 2
> },
> "http\_types": {
> "security4": 2
> }
> }
> }
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 8:52am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/9 "2017-12-26T08:52:32Z")
It doesn't look like you have a crazy amount of shards, which can often cause these kind of problems. How many shards are still UNASSIGNED?
Is there anything in the logs?
Do you have _minimum\_master\_nodes_ set to 2?
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:09am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/11 "2017-12-26T09:09:09Z")
> curl -s [http://localhost:9200/\_cat/shards](http://localhost:9200/_cat/shards) | grep UNASS | wc -l
> 169
> GET /\_nodes/\_master
> {
> "\_nodes": {
> "total": 1,
> "successful": 1,
> "failed": 0
> },
> "cluster\_name": "prod",
> "nodes": {
> "dEcFJ6h\_QfyKNr\_N7362QQ": {
> "name": "dEcFJ6h",
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 9:09am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/12 "2017-12-26T09:09:18Z")
@radi Please open a separate thread for your completely unrelated question.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 9:10am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/13 "2017-12-26T09:10:55Z")
> [@Christian\_Dahlqvist](#):
>
> Do you have 'minimum\_master\_nodes' set to 2?
Can you please check this in your elasticsearch.yml config file?
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:12am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/14 "2017-12-26T09:12:15Z")
No i dont have
> root@04elasticsearch:/home/ubuntu# cat /etc/elasticsearch/elasticsearch.yml | grep -i master
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:13am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/15 "2017-12-26T09:13:18Z")
> cluster.name: prod
> path.data: /vol
> path.logs: /vol/logs
> network.host: 0.0.0.0
> http.port: 9200
> discovery.zen.ping.unicast.hosts: ["10......","10......"]
> xpack:
> security:
> authc:
> realms:
> native1:
> type: native
> order: 0
All my conf
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 9:18am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/16 "2017-12-26T09:18:55Z")
If you have 2 (or 3) master eligible nodes you need to set [minimum\_master\_nodes to 2 in order to avoid split-brain scenarios](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/modules-node.html#split-brain). This means that the cluster will go red as soon as one node is missing, which is the correct behaviour in order to prevent data loss. If you need the cluster to be able to operate with one node down or unavailable, you need a minimum of 3 master eligible modes (which allows a majority of modes to elect a master even with one mode missing).
You might therefore be having a split-brain scenario, preventing the shards to be found and allocated.
Fix this and see if that allows the shards to get allocated.
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:32am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/17 "2017-12-26T09:32:05Z")
> PUT \_cluster/settings
> {
> "transient": {
> "discovery.zen.minimum\_master\_nodes": 2
> }
> }
> {
> "acknowledged": true,
> "persistent": {},
> "transient": {
> "discovery": {
> "zen": {
> "minimum\_master\_nodes": "2"
> }
> }
> }
> }
But got the same result.
I found that some index have size parameter, but someone no.
> logstash-2017.12.08 2 p UNASSIGNED
Index which had size parameter i've tried to do reindex and looks like that helped.
But all others dont want to work at that way
ex.
> POST \_reindex
> {
> "source": {
> "index": "logstash-2017.12.08"
> },
> "dest": {
> "index": "logstash-2017.12.33"
> }
> }
> {
> "error": {
> "root\_cause": ,
> "type": "search\_phase\_execution\_exception",
> "reason": "all shards failed",
> "phase": "query",
> "grouped": true,
> "failed\_shards":
> },
> "status": 503
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 9:35am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/18 "2017-12-26T09:35:14Z")
What is the output of the [cat nodes API](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/cat-nodes.html)?
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:36am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/19 "2017-12-26T09:36:03Z")
> ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m node.role master name
> 10.3.2.39 9 98 6 0.07 0.30 0.81 mdi - dEcFJ6h
> 10.3.2.45 9 95 6 0.07 0.34 0.65 mdi \* 7vuMfym
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 26, 2017, 9:39am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/20 "2017-12-26T09:39:42Z")
> [@Serg](#):
>
> I found that some index have size parameter, but someone no.
Not sure I understöd what you mean...
Now that both nodes are part of the same cluster, are shards getting allocated in the background? Do you see anything in the logs?
---
### Author:  [@Serg](https://discuss.elastic.co/u/Serg)
#### Post date: [December 26, 2017, 9:42am UTC](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154/21 "2017-12-26T09:42:41Z")
normal shards have a column with size
> logstash-2017.11.14 0 p STARTED 3516457 3.2gb 10.3.2.45 7vuMfym
but UNASSIGNED dont have
> logstash-2017.12.17 1 p UNASSIGNED
[Next page](https://discuss.elastic.co/t/could-not-reassign-unassigned-shards-elasticsearch-5-6/113154.md?page=2)