No valid shards

Elastic Stack Elasticsearch
1

I'm having some issues with my index where it won't allocate any shards so I can't create an Index Pattern. I've looked into the Stats on the Index Management page, which gets:

  "_shards": {
    "total": 2,
    "successful": 0,
    "failed": 0
  }
}

I've also started debugging in dev:

"index" : "filebeat-7.14.0-2021.08.24-000001",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "PRIMARY_FAILED",
    "at" : "2021-08-24T11:50:00.607Z",
    "details" : "primary failed while replica initializing",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "50Eq1WDAQjWVqCHjDyV1jg",
      "node_name" : "aegis-es-default-3",
      "transport_address" : "10.112.4.13:9300",
      "node_attributes" : {
        "k8s_node_name" : "gke-elastic-hot-a4cab97d-t91m",
        "ml.machine_memory" : "61606215680",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "2147483648",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "replica_after_primary_active",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        },
        {
          "decider" : "throttling",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        }
      ]
    }

Error: [object Object]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [org.elasticsearch.action.fieldcaps.FieldCapabilitiesIndexRequest/unset]
    at https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:376579
    at async index_patterns_IndexPatternsService.refreshFieldSpecMap (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:529228)
    at async index_patterns_IndexPatternsService.initFromSavedObject (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:532115)
    at async index_patterns_IndexPatternsService.createSavedObject (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:536335)
    at async index_patterns_IndexPatternsService.createAndSave (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:535578)
    at async https://localhost:5601/42747/bundles/plugin/indexPatternManagement/kibana/indexPatternManagement.chunk.1.js:1:559076
    at async https://localhost:5601/42747/bundles/plugin/indexPatternManagement/kibana/indexPatternManagement.chunk.1.js:1:548920
2

Hello, do you have at least one data node in your cluster? Is your cluster healthy?

3

Yes, I've got 5 node pools with all healthy nodes

4

You should check the primary shards of filebeat-7.14.0-2021.08.24-000001, i.e. if they are allocated / if not, why not.

Can you create any indices at all? (E.g. with a simple PUT myindex call in Kibana dev console.)

5

I can't create any indices, same error comes up. I've got 5 node pools attached to my cluster

6

Can you post the output of these commands?

GET _cat/nodes?v
GET _cluster/health?pretty
7

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.112.4.16 50 4 0 0.12 0.09 0.03 m - x-es-master-0
10.112.4.13 45 4 0 0.12 0.09 0.03 cdfhilmrstw - x-es-default-3
10.112.2.12 39 10 2 0.05 0.05 0.19 cdfhilmrstw - x-es-default-4
10.112.0.13 17 10 2 0.16 0.15 0.17 cdfhilmrstw - x-es-default-5
10.112.1.17 65 10 2 1.27 1.04 0.55 cdfhilmrstw - x-es-default-6
10.112.5.17 11 4 0 0.17 0.23 0.25 cdfhilmrstw - x-es-default-2
10.112.4.14 10 4 0 0.12 0.09 0.03 cdfhilmrstw - x-es-default-8
10.112.2.14 7 9 2 0.05 0.05 0.19 m - x-es-master-2
10.112.1.16 19 10 2 1.27 1.04 0.55 cdfhilmrstw * x-es-default-0
10.112.6.4 11 5 0 0.06 0.24 0.25 cdfhilmrstw - x-es-default-7
10.112.0.15 11 9 2 0.16 0.15 0.17 m - x-es-master-1

8 
{
  "cluster_name" : "aegis",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 11,
  "number_of_data_nodes" : 8,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 88.0
}
9

Is shard allocation enabled? In GET _cluster/settings you should have

"cluster.routing.allocation.enable" should be either unset or set to 'all'.

Also if you have a primary shard unassigned, you should try to see why that is unallocated. (

GET _cluster/allocation/explain?pretty

).

1 Like
10

It was an issue with the k8s cluster not having enough resources. Thanks!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.