Priority On Data Nodes In Elastic Search

Elastic Stack Elasticsearch
1

We have a self-hosted cluster on Kubernetes (EKS) with 1 master node and 2 master-data nodes. To get the performance, we have used NVMe instances. But if both the data node dies, the cluster gets corrupt and we have a loss of data.

However, we can use EBS volume to prevent data loss and prevent the cluster from getting corrupt.

Is there a way to use EBS attached node only in the time of disaster?

1 Like
2

How would that work? How would the data get written to the EBS volume?

If you require data to be written to EBS storage and do not want to lose data your nodes need to be configured to use EBS storage.

3

That was exactly my question. I would configure 1 of the node to use EBS volume and the rest of the node to use local storage(using NVMe instance).

But since kubernetes sends the request in round robin fashion, the request going to EBS volume will be slow as compared to local storage. I want the request to go to NVMe instances only in case of disaster.

4

I'm also stuck in a similar problem.

I have 2 data nodes based on ephemeral (local NVMes) storage and 1 on EBS. Can I define priority for the index leader? I want to use the NVMe nodes (higher priority) to serve traffic, but if they're down (the data is wiped out), then fallback to EBS until data is replicated to any NVMe data nodes.

5

I don't think ES supports anything like this, basically tiered storage on a single index (rather than between indexes like ILM does). Maybe others have more insight than me.

First problem is you'll feel the slower speed on indexing - you'd just have a replica count of 2 so there are 3 shards of each index, but that slower 3rd replica will block the high index rates possible on the NVMe.

Also searches would go to all shards unless you did some fancy routing (which I don't think is even possible). So hard to see how you can do this for reliability - do you have high risk of losing both NVMe nodes? If so, best to get another one.

6

That's correct.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.