Yellow heath, Unassigned shards after restart

Elastic Stack Elasticsearch
1

Hi Guys, Im sure this is a simple one, but i cant seem to resolve.

My ES node ran our of JVM space. So i modified its upper limit and restarted ES as ive done many times.
This time round, it refuses to go Green. Its been yellow now for two days.

The ONLY thing i can find is two shards not being assigned. Logs are empty of anything relevant.

Any thoughts?
Thanks in advance!

{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 611,
  "active_shards" : 611,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 4,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.34959349593497
}

image
image2679×642 51.9 KB

2 
{

* "state": "UNASSIGNED",
* "primary": false,
* "node": null,
* "relocating_node": null,
* "shard": 0,
* "index": "metrics-index_pattern_placeholder",
* "recovery_source": {
  * "type": "PEER"},
* "unassigned_info": {
  * "reason": "CLUSTER_RECOVERED",
  * "at": "2020-09-22T08:19:35.187Z",
  * "delayed": false,
  * "allocation_status": "no_attempt"}

}
3

More info

{

* "index": "logs-index_pattern_placeholder",
* "shard": 0,
* "primary": false,
* "current_state": "unassigned",
* "unassigned_info": {
  * "reason": "CLUSTER_RECOVERED",
  * "at": "2020-09-22T08:19:35.187Z",
  * "last_allocation_status": "no_attempt"},
* "can_allocate": "no",
* "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
* "node_allocation_decisions": [
  * {
    * "node_id": "WMC9UvMnTlWXGZGIjHuJqA",
    * "node_name": "siem-es-01",
    * "transport_address": "127.0.0.1:9300",
    * "node_attributes": {
      * "ml.machine_memory": "8348577792",
      * "xpack.installed": "true",
      * "transform.node": "true",
      * "ml.max_open_jobs": "20"},
    * "node_decision": "no",
    * "deciders": [
      * {
        * "decider": "same_shard",
        * "decision": "NO",
        * "explanation": "a copy of this shard is already allocated to this node [[logs-index_pattern_placeholder][0], node[WMC9UvMnTlWXGZGIjHuJqA], [P], s[STARTED], a[id=0rZvrNBASBadFXKGJQOOdw]]"}]}]

}
4

Do the indices with unassigned shard by any chance have a replica shard configured?

5

Still new to it.. where would i check that?

It does appear that one shard is allocated and fine, but there is another that isnt.. so that does sound and make sense..

6

Check the index settings. As you have only one node, replica shards will never get allocated as Elasticsearch never allocates them on the same node as the primary.

1 Like
7

Thanks,
Seems to fix the issue updating the index setting to 0 replicas..

No idea why it randomly decided to add an index with 1 replica on a single node cluster.

Cheers again

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.