No valid shards

I'm having some issues with my index where it won't allocate any shards so I can't create an Index Pattern. I've looked into the Stats on the Index Management page, which gets:

  "_shards": {
    "total": 2,
    "successful": 0,
    "failed": 0
  }
}

I've also started debugging in dev:

"index" : "filebeat-7.14.0-2021.08.24-000001",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "PRIMARY_FAILED",
    "at" : "2021-08-24T11:50:00.607Z",
    "details" : "primary failed while replica initializing",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "50Eq1WDAQjWVqCHjDyV1jg",
      "node_name" : "aegis-es-default-3",
      "transport_address" : "10.112.4.13:9300",
      "node_attributes" : {
        "k8s_node_name" : "gke-elastic-hot-a4cab97d-t91m",
        "ml.machine_memory" : "61606215680",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "2147483648",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "replica_after_primary_active",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        },
        {
          "decider" : "throttling",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        }
      ]
    }
Error: [object Object]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [org.elasticsearch.action.fieldcaps.FieldCapabilitiesIndexRequest/unset]
    at https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:376579
    at async index_patterns_IndexPatternsService.refreshFieldSpecMap (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:529228)
    at async index_patterns_IndexPatternsService.initFromSavedObject (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:532115)
    at async index_patterns_IndexPatternsService.createSavedObject (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:536335)
    at async index_patterns_IndexPatternsService.createAndSave (https://localhost:5601/42747/bundles/plugin/data/kibana/data.plugin.js:1:535578)
    at async https://localhost:5601/42747/bundles/plugin/indexPatternManagement/kibana/indexPatternManagement.chunk.1.js:1:559076
    at async https://localhost:5601/42747/bundles/plugin/indexPatternManagement/kibana/indexPatternManagement.chunk.1.js:1:548920

Hello, do you have at least one data node in your cluster? Is your cluster healthy?

Yes, I've got 5 node pools with all healthy nodes

You should check the primary shards of filebeat-7.14.0-2021.08.24-000001, i.e. if they are allocated / if not, why not.

Can you create any indices at all? (E.g. with a simple PUT myindex call in Kibana dev console.)

I can't create any indices, same error comes up. I've got 5 node pools attached to my cluster

Can you post the output of these commands?

GET _cat/nodes?v
GET _cluster/health?pretty

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.112.4.16 50 4 0 0.12 0.09 0.03 m - x-es-master-0
10.112.4.13 45 4 0 0.12 0.09 0.03 cdfhilmrstw - x-es-default-3
10.112.2.12 39 10 2 0.05 0.05 0.19 cdfhilmrstw - x-es-default-4
10.112.0.13 17 10 2 0.16 0.15 0.17 cdfhilmrstw - x-es-default-5
10.112.1.17 65 10 2 1.27 1.04 0.55 cdfhilmrstw - x-es-default-6
10.112.5.17 11 4 0 0.17 0.23 0.25 cdfhilmrstw - x-es-default-2
10.112.4.14 10 4 0 0.12 0.09 0.03 cdfhilmrstw - x-es-default-8
10.112.2.14 7 9 2 0.05 0.05 0.19 m - x-es-master-2
10.112.1.16 19 10 2 1.27 1.04 0.55 cdfhilmrstw * x-es-default-0
10.112.6.4 11 5 0 0.06 0.24 0.25 cdfhilmrstw - x-es-default-7
10.112.0.15 11 9 2 0.16 0.15 0.17 m - x-es-master-1

{
  "cluster_name" : "aegis",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 11,
  "number_of_data_nodes" : 8,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 88.0
}

Is shard allocation enabled? In GET _cluster/settings you should have

"cluster.routing.allocation.enable" should be either unset or set to 'all'.

Also if you have a primary shard unassigned, you should try to see why that is unallocated. (

GET _cluster/allocation/explain?pretty

).

1 Like

It was an issue with the k8s cluster not having enough resources. Thanks!