Elasticsearch keep migrating shards out of one of my data node

Elastic Stack Elasticsearch
,
4

This is weird then, the node was full, with a similar amount of shards and then Elastic started to empty it out?

Do you have anything in the logs for this node? Like it went out of the cluster for a couple of minutes?

I would set all the allocation settings that you changed to their default values, there is an open issue about changing the cluster_concurrent_rebalance from the default could lead to redundant shard movement which can increase the time the cluster would reach the balance.

These settings:

        "allocation": {
          "node_concurrent_incoming_recoveries": "4",
          "cluster_concurrent_rebalance": "10",
          "node_concurrent_recoveries": "4",
          "node_initial_primaries_recoveries": "4",
          "enable": "all",
          "node_concurrent_outgoing_recoveries": "4"
        }

Also, some of your nodes on rtp2 have reached or are close to reach the first disk watermark, adding this to the non-default allocation settings it could make a lot of redundant shard movement.

1 Like