Create Horizontal pod autoscaling for elasticsearch in kubernetes (eck operator)

Elastic Stack Elasticsearch
1

ECK = 1.0.0 ga
Cloud= EKS

Kibana version : 7.5.2

Elasticsearch version : 7.5.2

EKS: 1.15

Please provide details for Horizontal pod autoscaling for elasticsearch in kubernetes (eck operator)

2

Autoscaling Elasticsearch can be tricky as it can hold a lot of data and adding or removing nodes will cause data to need to be redistributed. What is your use case? How much data do you expect to have in Elasticsearch? What is the driver behind using autoscaling?

3

Data is 100gb per node. and we have 3 nodes. The driver is based on cpu,mem.

4

So you have a fixed data set where all nodes hold a complete copy of all shards?

Will you be scaling up as well as down?

Have you verified that you are limited by CPU and/or RAM rather than e.g. disk I/O given that your data set per node is large enough to likely not be all cached? The reason I am asking this is that adding a new node will add additional load to the cluster as large amounts of data need to be copied. Doing this when your cluster is at peak load (or close to it) could easily reduce performance when you least can afford it.

What instance types are you using? What type of storage?

5

So you have a fixed data set where all nodes hold a complete copy of all shards?

YES

Will you be scaling up as well as down?

YES

Have you verified that you are limited by CPU and/or RAM rather than e.g. disk I/O given that your data set per node is large enough to likely not be all cached? The reason I am asking this is that adding a new node will add additional load to the cluster as large amounts of data need to be copied. Doing this when your cluster is at peak load (or close to it) could easily reduce performance when you least can afford it.

We will have autoscaling at safe limits. i.e if we reach average cpu 70% utilisation or
average mem 70%. we need to scale. in that way we can scale without adding much strain.

Instance types will be aws 5th gen & storage ebs

Anyway this is for testing, can you share details on hpa for this.

6

Given that autoscaling Elasticsearch as far as I know generally is not recommended I am not sure there is any support for this built inb, so will have to leave this for others to respond to.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.