Elasticsearch 7.2.0 Cluster - Performance Recovery Help

Elastic Stack Elasticsearch
1

I am new to elasticsearch management and have some issues with a cluster I am working with.

Currently, the cluster is recovering from a node failure, but I would like to stop the recovery and delete a handful of indices. Then reload the bad index myself.
Data loss is not important. Just want to get the cluster back to green as quickly as possible even if the indexes are all deleted or something.

Any time I try to delete an index from the cluster the cluster returns a 503 error. I assume it's because it's busy trying to initialize the unassigned shards. That process is taking too long and I would just like to stop it, but I'm not sure how.

Should I just let the recovery process finish or is there another way to get the cluster back to green?

  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 194,
  "active_shards" : 194,
  "relocating_shards" : 0,
  "initializing_shards" : 8,
  "unassigned_shards" : 243,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 23,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 22808480,
  "active_shards_percent_as_number" : 43.59550561797752

the cluster allocation

{
  "index" : "inventory",
  "shard" : 3,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2021-03-17T01:54:32.146Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "throttled",
  "allocate_explanation" : "allocation temporarily throttled",
  "node_allocation_decisions" : [
    {
      "node_id" : "Fu_5cfOGTl6wZ3ucWPQWZw",
      "node_name" : "node2",
      "transport_address" : "x",
      "node_attributes" : {
        "ml.machine_memory" : "137391284224",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "throttled",
      "deciders" : [
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [4], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    },
    {
      "node_id" : "XjS2M3lMSIyqKghVnbPtTg",
      "node_name" : "node1",
      "transport_address" : "x",
      "node_attributes" : {
        "ml.machine_memory" : "137391284224",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[inventory][3], node[XjS2M3lMSIyqKghVnbPtTg], [P], s[STARTED], a[id=PCmG3kp7SRCvmosVy5yecA]]"
        },
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [4], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    }
  ]
}```

Again, not sure what I really need/want, but I would like to delete some of the indexes from the cluster, but the cluster had node failure.

-Owen
2

Welcome to our community! :smiley:

The only way I can think of is to disable allocation, delete any currently allocated indices, then enable allocation again.

You cannot delete indices that are not allocated.

3

I don't think that's the case, you should be able to delete indices whether they're allocated or not.

What exactly is the 503 response? If it's a timeout, you can increase the timeouts with something like ?timeout=1d&master_timeout=1d.

Also 7.2.0 is really quite old, long past EOL, and there have been improvements in this area. You should upgrade as a matter of some urgency.

4

Oh, well that's great to know! :smiley:

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.