# Internal monitoring indices shards in UNASSIGNED state **URL:** \ **Category:** Elasticsearch\ **Created:** [December 15, 2022, 3:06am UTC](https://discuss.elastic.co/t/internal-monitoring-indices-shards-in-unassigned-state/321247 "2022-12-15T03:06:40Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [December 15, 2022, 3:06am UTC](https://discuss.elastic.co/t/internal-monitoring-indices-shards-in-unassigned-state/321247/1 "2022-12-15T03:06:40Z")
Hi, We have an 8.x ES cluster running on Elastic cloud, All seems to be working ok except that a few internal monitoring indices have one shard marked as UNASSIGNED and this causes the cluster state to be YELLOW, all the time... I followed this Guide to troubleshoot: > **[Diagnose unassigned shards | Elasticsearch Guide \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/diagnose-unassigned-shards.html)** Get info about shards - find which ones are unassigned: ```auto GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state ``` Response as of Dec 14, 9:33PM EST: ```auto index shard prirep state node unassigned.reason .ds-.monitoring-es-8-mb-2022.12.14-000003 0 r UNASSIGNED INDEX_CREATED .ds-.monitoring-kibana-8-mb-2022.12.14-000003 0 r UNASSIGNED INDEX_CREATED ``` Get more details about the allocation for the specific index/shard: ```auto GET _cluster/allocation/explain { "index": ".ds-.monitoring-es-8-mb-2022.12.14-000003", "shard": 0, "primary": false } ``` Result: ```auto { "index": ".ds-.monitoring-es-8-mb-2022.12.14-000003", "shard": 0, "primary": false, "current_state": "unassigned", "unassigned_info": { "reason": "INDEX_CREATED", "at": "2022-12-14T21:45:50.221Z", "last_allocation_status": "no_attempt" }, "can_allocate": "no", "allocate_explanation": "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.", "node_allocation_decisions": [ { "node_id": "hYTLxs3cQwSSQNRWNCVHJg", "node_name": "instance-0000000001", "transport_address": "10.xxx:19240", "node_attributes": { "logical_availability_zone": "zone-0", "server_name": "instance-0000000001.aa7f0380c25d414f810f9a23d173130b", "availability_zone": "us-east4-a", "xpack.installed": "true", "data": "warm", "instance_configuration": "gcp.es.datawarm.n2.68x10x190", "region": "unknown-region" }, "node_decision": "no", "weight_ranking": 1, "deciders": [ { "decider": "data_tier", "decision": "NO", "explanation": "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier" } ] }, { "node_id": "RR_pUkQdRR2HqwjMlNhtVg", "node_name": "instance-0000000000", "transport_address": "10.xxx:19870", "node_attributes": { "region": "unknown-region", "instance_configuration": "gcp.es.datahot.n2.68x10x45", "server_name": "instance-0000000000.aa7f0380c25d414f810f9a23d173130b", "data": "hot", "xpack.installed": "true", "logical_availability_zone": "zone-0", "availability_zone": "us-east4-a" }, "node_decision": "no", "weight_ranking": 2, "deciders": [ { "decider": "same_shard", "decision": "NO", "explanation": "a copy of this shard is already allocated to this node [[.ds-.monitoring-es-8-mb-2022.12.14-000003][0], node[RR_pUkQdRR2HqwjMlNhtVg], [P], s[STARTED], a[id=fxfbSsCnTq-DGvTI4laIPg]]" } ] } ] } ``` This part seems to offer a promising reason: ```auto "explanation": "index has a preference for tiers [data_hot] and node does not meet the required [data_hot] tier" ``` However, I'm not sure how to solve this - since I am not controlling templates or ILMs for these internal indices and do not set some custom policies for shard allocations ... Whatever is the default policy - it should be used as is. My cluster config: -- 2 nodes -- 1 availability zone -- Hot and Warm storage only, no cold Any advise on how to troubleshoot this further would be very appreciated! Thank you! Marina ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 15, 2022, 6:34am UTC](https://discuss.elastic.co/t/internal-monitoring-indices-shards-in-unassigned-state/321247/2 "2022-12-15T06:34:21Z")
Hi @ppine7 Are you saying you only have one hot and one warm node in one zone. By default all the system, indices and monitoring indices (basically as well as any other defaults) expect to have one primary plus one replica shard. If you only have one hot node you have no place to put the replica shard because of primary and replica will not be on the same node. So your choice is to set replicas to zero or live with the yellow cluster or add another hot node, preferably in a different zone. Same will go for warm when you get there... Basically when you only have one zone by a default your cluster's always going to be yellow unless you set replica's to zero on every index. It's a way of telling you that you don't have any high availability. If you lose one node, you're down. Hopefully that makes sense. ---
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [December 15, 2022, 4:48pm UTC](https://discuss.elastic.co/t/internal-monitoring-indices-shards-in-unassigned-state/321247/3 "2022-12-15T16:48:49Z")
Thank you, @stephenb ! Yes, unfortunately due to costs we decided to go with the cheapest configuration of the ES cluster, just 1 zone, since the data is not critical .... I suspected that was the case... not enough nodes for all replicas. But I am still puzzled by a few points: -- if the index template fo rthose internal indices requires two shards - why not ALL of them are UNASSIGNED? but only some? What I mean is: There are a few indices that were created for monitoring already, I'm assuming form the smae template, right? But when I check indices - I see that only the last index from each "Type/template" is in th eYELLOW state with unassigned shards.... Here is an example: ![index-stats](https://us1.discourse-cdn.com/elastic/original/3X/0/4/04d891b6fb079b7844dda76547bd0592174d6e80.jpeg) so, for example , this index is GREEN: [.ds-.monitoring-kibana-8-mb-2022.12.11-000002](https://tt-ibc-prod-elastic.kb.us-east4.gcp.elastic-cloud.com:9243/app/monitoring#/elasticsearch/indices/.ds-.monitoring-kibana-8-mb-2022.12.11-000002?_g=(cluster_uuid:qf7EJYmGSlyFFrnqePAb7A,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))) but the next one of the same type is YELLOW: [.ds-.monitoring-kibana-8-mb-2022.12.14-000003](https://tt-ibc-prod-elastic.kb.us-east4.gcp.elastic-cloud.com:9243/app/monitoring#/elasticsearch/indices/.ds-.monitoring-kibana-8-mb-2022.12.14-000003?_g=(cluster_uuid:qf7EJYmGSlyFFrnqePAb7A,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))) Why is the first one has 0 unassigned shards but the second one - 1? Should not they both be created i exactly the same way? So I also checked settings for indices to see which ones have replication set to 1: `GET .monitoring*/_settings/index.number_of_replicas` and see that all above patterns have RF = 1 (not 0): ```auto ".ds-.monitoring-es-8-mb-2022.12.11-000002": { "settings": { "index": { "number_of_replicas": "1" } } }, ".monitoring-beats-7-2022.12.15": { "settings": { "index": { "number_of_replicas": "0" } } }, ".ds-.monitoring-es-8-mb-2022.12.14-000003": { "settings": { "index": { "number_of_replicas": "1" } } }, ".ds-.monitoring-kibana-8-mb-2022.12.14-000003": { "settings": { "index": { "number_of_replicas": "1" } } }, ".ds-.monitoring-kibana-8-mb-2022.12.11-000002": { "settings": { "index": { "number_of_replicas": "1" } } } } ``` still one index is GREEN the other is YELLOW - even though they are created form the same template (I assume). Further I noticed one more weird (to me) thing: the index and shard distribution is not equal across two nodes that I have: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a073717e1cac4f9bbfb95edeb90946a5b751b2bc.png) most of the shards are on the node 0000, and only 2 shards are on the node 0001. Funnily enough, those 2 shards on the node 0001 are the GREEN shards from the indices I showed above: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/8/286cf7e5a36d1a85bc613668b4214f5590fc6e22.png) while the later shards of the same indices are YELLOW on the node 0000 .... ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/4/740a8a51166fd6f1cead9741b3f084b1ac38fecb.png) something seems wrong with this distribution ๐Ÿ™‚ Thank you!! Marina ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 15, 2022, 7:54pm UTC](https://discuss.elastic.co/t/internal-monitoring-indices-shards-in-unassigned-state/321247/4 "2022-12-15T19:54:30Z")
Hi @ppine7 Back to Basics... 1. If you only have 1 node per tier unless replicas are 0 you will always be yellow in a properly configured cluster because Primary and Replicas can not be on the same node. 2. An Index is only on 1 Data Tier at a time. 3. Primaries and replicas for the same Index live on the same data tier, i.e. they do not span data tiers, and are not allowed on the same node. Solution : Set All Replicas to 0 or Have 2 Nodes per tier tier. I really can not debug all this ..But general is what I believe is happening is there are some issues with perhaps some of the with respect to data\_content, data\_hot, data\_warm and so some of those templates is allowing some of the Primary / Replica to span tiers which is not normal, maybe its a bug or some other side effect... > [@ppine7](#): > > something seems wrong with this distribution ๐Ÿ™‚ Not sure why you think that is wrong... it is showing all the primaries on 1 node... since you only have 1 hot node that is exactly what I would expect..... If you set ALL indices to 0 replicas your cluster would go green... but as soon as a template creates a new index with a replica it will go yellow... tl;dr if you are only in 1 zone get used to yellow cluster ๐Ÿ™‚ or fix all the templates and indices! > [@ppine7](#): > > ```auto > "xpack.installed": "true", > "data": "warm",