We are running a 17 node cluster with 10 dedicated hot nodes, 3 master nodes and 4 dedicated warm nodes.
We have an index with 5 primary shards and a replication factor of 1 that writes only to the hot nodes, which is dictated by a template.
We have implemented a lifecycle policy that when there is 50Gb of storage in the primary shards, it rolls over to a shrink index reducing the shard count to 1 primary and replica shard. This index is on the warm nodes.
Before it is shrunk, it moves the index onto the warm node, reallocates 1 of each shard to a single node then begins the shrink process.
This all works, but the issue is we have a high ingest rate ~20k second on the hot nodes and the recovery speed is only 40Mb/s per recovery happening on a node, which is observed by running nload -u M on the nodes. This causes the cluster to struggle as it's not finishing the previous steps before the new shrink index is getting created and the index getting moved to the warm nodes.
An example would be:
warm-0 -> warm-1
warm-2 -> warm-1
warm-1 would have an incoming rate of ~80Mb/s
warm-3 -> warm-4
warm-4 would have an incoming rate of ~40Mb/s
Even when both of these are the same size VM. The first example suggests that the VM can handle at least 80Mb/s - however, this is unattainable on the warm-4.
Our cluster configuration settings are as follows:
{
"persistent" : {
"cluster" : {
"routing" : {
"allocation" : {
"cluster_concurrent_rebalance" : "2",
"node_concurrent_recoveries" : "2"
}
}
},
"indices" : {
"recovery" : {
"max_bytes_per_sec" : "1024mb",
"max_concurrent_file_chunks" : "5"
}
}
}
}
Our elasticsearch.yml is as follows:
cluster.name: "cluster"
node.name: "warm-2"
path.logs: /var/log/elasticsearch
path.data: /datadisks/disk1/elasticsearch/data
discovery.seed_hosts: ["master-0","master-1","master-2"]
cluster.initial_master_nodes: ["master-0","master-1","master-2"]
node.master: false
node.data: true
network.host: [_site_, _local_]
node.max_local_storage_nodes: 1
node.attr.fault_domain: 0
node.attr.update_domain: 12
node.attr.box_type: warm
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
An image of one the nodes pushing a shard into another:
Basically we're looking for the bottleneck.