# Fresh cluster all shards are unavailable **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [December 2, 2019, 4:31pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216 "2019-12-02T16:31:49Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 2, 2019, 4:31pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/1 "2019-12-02T16:31:49Z")
Hi, When my cluster is started, his status is in yellow: > { > "cluster\_name" : "datawarehouse", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 6, > "number\_of\_data\_nodes" : 0, > "active\_primary\_shards" : 0, > "active\_shards" : 0, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 9, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 0.0 > } Also in parallel I check the kibana logs (because the service can't start), the error is: > {"type":"log","@timestamp":"2019-12-02T15:58:39Z","tags":["security","error"],"pid":6,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [unavailable\_shards\_exception] at least one primary shard for the index [.security-7] is unavailable"} I check my shards: > curl -k -u "elastic:xxxxx" "[https://datawarehouse-es-http:9200/\_c](https://datawarehouse-es-http:9200/_c) > at/shards?h=index,shard,prirep,state,unassigned.reason" --silent > .security-7 0 p UNASSIGNED INDEX\_CREATED > .kibana\_task\_manager\_1 0 p UNASSIGNED INDEX\_CREATED > .kibana\_task\_manager\_1 0 r UNASSIGNED INDEX\_CREATED > .kibana\_1 0 p UNASSIGNED INDEX\_CREATED > .kibana\_1 0 r UNASSIGNED INDEX\_CREATED > .apm-agent-configuration 0 p UNASSIGNED INDEX\_CREATED > .apm-agent-configuration 0 r UNASSIGNED INDEX\_CREATED Then I check the cluster allocation: > curl -k -u "elastic:xxxxx" "[https://datawarehouse-es-http:9200/\_c](https://datawarehouse-es-http:9200/_c) > luster/allocation/explain?pretty" > { > "index" : ".kibana\_1", > "shard" : 0, > "primary" : true, > "current\_state" : "unassigned", > "unassigned\_info" : { > "reason" : "INDEX\_CREATED", > "at" : "2019-12-02T15:34:31.969Z", > "last\_allocation\_status" : "no\_attempt" > }, > "can\_allocate" : "no", > "allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes" > } Also I found this error in the elasticsearch logs: org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable I check the cluster settings: > curl -k -u "elastic:xxxxx" [https://datawarehouse-es-http:9200/\_cl](https://datawarehouse-es-http:9200/_cl) > uster/settings?pretty > { > "persistent" : { }, > "transient" : { > "cluster" : { > "routing" : { > "allocation" : { > "exclude" : { > "\_name" : "none\_excluded" > } > } > } > } > } > } ECK version: 1.0.0-beta elasticsearch version: 7.4.2 elasticsearch config: [eck elastic config · GitHub](https://gist.github.com/Dudesons/f4af00f15790175e3f360577b8f04f15) (it's the config of 1 nodes) cluster: 3 master & 3 data nodes I tried to boot a new cluster, the problem persists If someone can help me to understand / fix the issue it would be awesome. ---
**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\ **Post date:** [December 2, 2019, 7:01pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/2 "2019-12-02T19:01:51Z")
What is the status of the ES resources? `kubectl get elasticsearch` (or `describe`) Also, the explain API can provide useful information for why shards are not allocated: [https://www.elastic.co/guide/en/elasticsearch/reference/6.0/cluster-allocation-explain.html#\_explain\_api\_response](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/cluster-allocation-explain.html#_explain_api_response) ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [December 3, 2019, 8:06am UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/3 "2019-12-03T08:06:00Z")
@dg_hivebrite are you using PersistentVolumes? Can you share the Elasticsearch yaml manifest? I'm wondering if one of your volume hosting data has been lost. ---
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 3, 2019, 10:38am UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/4 "2019-12-03T10:38:46Z")
@Anya_Sabo the kubectl get elasticsearch: > NAME HEALTH NODES VERSION PHASE AGE > datawarehouse yellow 6 7.4.2 Ready 19h And the describe: [es\_decribe.yaml · GitHub](https://gist.github.com/Dudesons/8dc56e0813ba1030302aa00af1ab2203) ---
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 3, 2019, 10:47am UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/5 "2019-12-03T10:47:29Z")
@sebgl Yes I'm using persitent volume. Acually the cluser is managed in GKE. The output of my pv: > NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE > pvc-022edc16-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-master-europe-west1-a-0 standard 19h > pvc-02dc04aa-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-master-europe-west1-b-0 standard 19h > pvc-03745ee2-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-master-europe-west1-c-0 standard 19h > pvc-03ed030a-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-data-europe-west1-a-0 standard 19h > pvc-04639049-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-data-europe-west1-b-0 standard 19h > pvc-04da5b0d-1519-11ea-9b78-4201c0a8000a 10Gi RWO Delete Bound default/elasticsearch-data-datawarehouse-es-data-europe-west1-c-0 standard 19h You want the yaml file before it send to kubernetes or an output of an object of kubernetes ? ---
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 3, 2019, 11:02am UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/6 "2019-12-03T11:02:55Z")
I don't understand why on the allocation explain call there is no nodes: > curl "[https://datawarehouse-es-http:9200/\_cluster/alloc](https://datawarehouse-es-http:9200/_cluster/alloc) > ation/explain?pretty&include\_disk\_info=true&include\_yes\_decisions=true" > { > "index" : ".kibana\_1", > "shard" : 0, > "primary" : true, > "current\_state" : "unassigned", > "unassigned\_info" : { > "reason" : "INDEX\_CREATED", > "at" : "2019-12-02T15:34:31.969Z", > "last\_allocation\_status" : "no\_attempt" > }, > "cluster\_info" : { > "nodes" : { }, > "shard\_sizes" : { }, > "shard\_paths" : { } > }, > "can\_allocate" : "no", > "allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes" > } ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [December 3, 2019, 12:13pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/7 "2019-12-03T12:13:43Z")
In the ES config I can see: ```auto cluster.routing.allocation.awareness.attributes: all ``` The value here should match one of the existing node attribute. Based on the rest of the Elasticsearch spec I can see you're using the attribute `zone` to distinguish group of nodes. You probably need to change the configuration to: ```auto cluster.routing.allocation.awareness.attributes: zone ``` ---
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 3, 2019, 1:28pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/8 "2019-12-03T13:28:53Z")
Ok, so I tried the change it doesn't change anything. Then I trash my cluster and reboot a new one without `cluster.routing.allocation.awareness.attributes`, the cluster is yellow with the same issue ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [December 3, 2019, 1:43pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/9 "2019-12-03T13:43:32Z")
Looking at your cluster again: can you double check it has at least one data node? I can see 3 master nodes, and another master node with: ```auto Config: cluster.routing.allocation.awareness.attributes: all node.attr.zone: europe-west1-a node.data: false node.master: true Count: 1 Name: data-europe-west1-a ``` which I guess was intended to have `node.data: true` looking at its name? ---
**Author:** ![dg\_hivebrite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dg_hivebrite/32/46359_2.png) [@dg\_hivebrite](https://discuss.elastic.co/u/dg_hivebrite)\ **Post date:** [December 3, 2019, 2:34pm UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/10 "2019-12-03T14:34:55Z")
Oh yes good catch, it was that the issue thank you very much 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:35am UTC](https://discuss.elastic.co/t/fresh-cluster-all-shards-are-unavailable/210216/11 "2022-11-04T07:35:45Z")