Elasticsearch disk based shard allocation tuning

Elastic Stack Elasticsearch
1

I am running 2 node Elasticsearch cluster v7.9.2
Ingesting data through APM-Server v7.15.0, creates ~50 new indices daily with pattern
apm-7.15.0-<service.name>-transaction-YYYY.MM.DD

Observing that all new primary shards of new indices are getting created on 1 node, even though the free disk space % is lesser than the other node.
How do I make the allocation of shards equally on both nodes ?
How can I assign the allocation for a certain index pattern on a particular node.

Below is the cluster setting

"persistent" : {
    "cluster" : {
      "routing" : {
        "rebalance" : {
          "enable" : "primaries"
        },
        "allocation" : {
          "node_concurrent_incoming_recoveries" : "5",
          "node_concurrent_recoveries" : "5",
          "disk" : {
            "watermark" : {
              "low" : "75%",
              "flood_stage" : "90%",
              "high" : "80%"
            }
          },
          "enable" : "primaries"
        }
      },
      "max_shards_per_node" : "3000"
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
2

Allocation is based on shard count, then total disk use as a percentage.
So what does _cat/allocation?v show?

You should really be running a newer version of Elasticsearch with APM at 7.15, ideally, the same versions.

That's likely to cause you issues.

3

Hi @warkolm , output for _cat/allocation?v is as below

shards disk.indices disk.used disk.avail disk.total disk.percent host            ip              node
   208        1.4tb     1.6tb        1tb      2.6tb           59 2.12.27.1  2.12.27.1  node1
   185        515gb   772.8gb    602.1gb      1.3tb           56 2.12.25.2  2.12.25.2  node2

if allocation is based on shard count, then in above case, if 50 new indices are created ;would all new shards get allocated to just one node ?

I have also tried the below settings in index template, and changed the weight setting in cluster - "cluster.routing.allocation.balance.index": "20.0"(tried value as 1,10,20)

    "number_of_routing_shards": "4",
    "number_of_shards": "2",
    "number_of_replicas": "0"

but still both shards of same index are allocated on same node. Can you help me resolve this so that I have 2 shards per index on separate nodes.

4

You probably want to look at using this, rather than altering cluster settings - Cluster-level shard allocation and routing settings | Elasticsearch Guide [7.16] | Elastic

5

Thanks @warkolm . awareness attribute worked for me.