Elasticsearch Constantly Reallocating Shards

Elastic Stack Elasticsearch
1

ES Version 5.6.2

For the past few weeks, every time I check ES Monitoring, twenty shards are being reallocated:
image

This is defined as the max in my cluster settings. I have it set higher than default as many of the machines I'm running on are random servers and not the most reliable. Hence if a node fails I want the cluster to dynamically reallocate shards. However no nodes have gone down in the past few weeks...

GET _cluster/settings

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "allow_rebalance": "always",
          "cluster_concurrent_rebalance": "20",
          "node_concurrent_recoveries": "20",
          "disk": {
            "watermark": {
              "low": "95%",
              "high": "98%"
            }
          }
        }
      }
    }
  },
  "transient": {}
}

When I look for an explanation on why a shard is being moved, I get nothing valuable:

GET /_cluster/allocation/explain
{
  "index": "nginx-2017.07.15s",
  "shard": 0,
  "primary": true
}

{
  "index": "nginx-2017.07.15s",
  "shard": 0,
  "primary": true,
  "current_state": "relocating",
  "current_node": {
    "id": "cW3TmEVIShSGxkCA8_zRew",
    "name": "mynode11",
    "transport_address": "192.168.100.100:9300",
    "attributes": {
      "ml.max_open_jobs": "10",
      "box_type": "warm",
      "ml.enabled": "true",
      "tag": "warm"
    }
  },
  "explanation": "the shard is in the process of relocating from node [mynode11] to node [mynode10], wait until relocation has completed"
}
2

Just a guess: maybe change cluster.routing.allocation.allow_rebalance back to the default indices_all_active?

3

Thanks for the suggestion. I updated cluster.routing.allocation.allow_rebalance to indices_all_active. My indices are all active, thus the relocation is still occurring.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.