ElasticSearch creating new index is unassigned even though it says that it can allocate

Elastic Stack Elasticsearch
1

We have a setup running ECK in AWS running version 8.10.2. Any attempt to create an index results in the shards not being assigned. I can manually assign them, but that is not a solution.

GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state

gives

index shard prirep state node unassigned.reason
products 0 p UNASSIGNED INDEX_CREATED
products 0 r UNASSIGNED INDEX_CREATED
products 1 p UNASSIGNED INDEX_CREATED
products 1 r UNASSIGNED INDEX_CREATED

Trying to understand this, I ran:

GET _cluster/allocation/explain
{
  "index": "products", 
  "shard": 1, 
  "primary": true 
}

Which gives:

"index": "products",
  "shard": 1,
  "primary": true,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "INDEX_CREATED",
    "at": "2024-02-15T07:35:09.361Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "yes",
  "allocate_explanation": "Elasticsearch can allocate the shard.",
  "target_node": {
    "id": "Q5b4XdeUQWOlhsmEfETL4g",
    "name": "elasticsearch-es-data-2",
    "transport_address": "10.2.240.60:9300",
    "attributes": {
      "ml.config_version": "10.0.0",
      "k8s_node_name": "ip-10-2-176-77.eu-central-1.compute.internal",
      "transform.config_version": "10.0.0",
      "xpack.installed": "true"
    }
  },
  "node_allocation_decisions": [{"node_decision": "yes",} ...omitted...

It says last_allocation_status": "no_attempt", "can_allocate": "yes" and "allocate_explanation": "Elasticsearch can allocate the shard."

I do not understand why it will not assign the shards. Running POST _cluster/reroute simply times out, and the nodes seem to have loads of storage.

GET _nodes/stats?metric=fs

"name": "elasticsearch-es-data-0",
"transport_address": "10.2.132.231:9300",
"host": "10.2.132.231",
"ip": "10.2.132.231:9300",
"roles": [
    "data",
    "ingest",
    "transform"
  ],
"attributes": {
  "k8s_node_name": "ip-10-2-149-147.eu-central-1.compute.internal",
  "transform.config_version": "10.0.0",
  "xpack.installed": "true",
  "ml.config_version": "10.0.0"
},
"fs": {
  "total": {
    "total_in_bytes": 9223372036853727000,
    "free_in_bytes": 9223372032458097000,
    "available_in_bytes": 9223372032458097000
  }
}

Maybe its not enough heap?

"heap_used_in_bytes":1382036912,
"heap_used_percent":64,
"heap_committed_in_bytes":2147483648,
"heap_max_in_bytes":2147483648,
"non_heap_used_in_bytes":294922728,
"non_heap_committed_in_bytes":335740928,

A simple PUT /test also times out, where the index is created, but shards are unassigned.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.