Clone index failing

Elastic Stack Elasticsearch
1

Hi all,
I'm getting an error when I try to clone my index called myindex into myindex_clone. I have set up a 3 node cluster on my machine, and I'm using the (Go) SDK client to call the CloneIndex() function. However I'm seeing the following error:

es1    | {"type": "server", "timestamp": "2021-12-08T17:56:59,344Z", "level": "WARN", "component": "o.e.i.c.IndicesClusterStateService", "cluster.name": "es-docker-cluster", "node.name": "es1", "message": "[myindex_clone][0] marking and sending shard failed due to [failed recovery]", "cluster.uuid": "9sYb7rmqQGmRXOLvyLxOxg", "node.id": "rzC7tCbeThypNSAY6KokIg" ,
es1    | "stacktrace": ["org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex_clone][0]: Recovery failed on {es1}{rzC7tCbeThypNSAY6KokIg}{K-JS8W07RkifXim9kF8cQg}{172.28.0.2}{172.28.0.2:9300}{cdhilmrstw}{ml.machine_memory=5175709696, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}",

The cluster status becomes red indefinitely. If I try curling the clone API endpoint it seems to work fine:

curl -X PUT "localhost:9200/myindex/_clone/myindex_clone-1"
{"acknowledged":true,"shards_acknowledged":true,"index":"myindex_clone-1"}

This is the result i'm seeing:

curl -X GET localhost:9200/_cat/indices\?format=json\&pretty
[
  {
    "health" : "green",
    "status" : "open",
    "index" : "myindex_clone-1",
    "uuid" : "CSufysmETAqrhNuGsQd3Ag",
    "pri" : "1",
    "rep" : "1",
    "docs.count" : "0",
    "docs.deleted" : "0",
    "store.size" : "416b",
    "pri.store.size" : "208b"
  },
  {
    "health" : "red",
    "status" : "open",
    "index" : "myindex_clone",
    "uuid" : "Fp_vdgXuSkGXvHYOFOmwiA",
    "pri" : "1",
    "rep" : "1",
    "docs.count" : null,
    "docs.deleted" : null,
    "store.size" : null,
    "pri.store.size" : null
  },
  {
    "health" : "green",
    "status" : "open",
    "index" : "myindex",
    "uuid" : "nFOgoOGwR0K4hpU-gEtyJQ",
    "pri" : "1",
    "rep" : "1",
    "docs.count" : "0",
    "docs.deleted" : "0",
    "store.size" : "416b",
    "pri.store.size" : "208b"
  }
]

If I do an erxplain on the cluster I get:

 curl -XGET "localhost:9200/_cluster/allocation/explain?pretty"
{
  "index" : "myindex_clone",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2021-12-08T17:56:59.273Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "19gFPBE1S-SNi0boE5llpQ",
      "node_name" : "es2",
      "transport_address" : "172.28.0.3:9300",
      "node_attributes" : {
        "ml.machine_memory" : "5175709696",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "replica_after_primary_active",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        },
        {
          "decider" : "throttling",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id" : "pER-Jz0OQ-WctEMRjahTQQ",
      "node_name" : "es3",
      "transport_address" : "172.28.0.4:9300",
      "node_attributes" : {
        "ml.machine_memory" : "5175709696",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "replica_after_primary_active",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        },
        {
          "decider" : "throttling",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id" : "rzC7tCbeThypNSAY6KokIg",
      "node_name" : "es1",
      "transport_address" : "172.28.0.2:9300",
      "node_attributes" : {
        "ml.machine_memory" : "5175709696",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 3,
      "deciders" : [
        {
          "decider" : "replica_after_primary_active",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        },
        {
          "decider" : "throttling",
          "decision" : "NO",
          "explanation" : "primary shard for this replica is not yet active"
        }
      ]
    }
  ]
}

But I'm unsure what that means exactly. I don't understand why the curl request would work while the request via the client SDK doesn't. Can anyone help me fixing this?

2

I found out why. Turns out i had set the current index to index.blocks.read_only when it should be index.blocks.write.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.