Hi
We are looking for scaling up and down Kibana/ES nodes automatically at runtime. I am aware about nodeSet feature in elasticsearch. Please let me know if HPA can be used as well ?
ECK Version in use: 1.1.0
Regards
Arun
Auto-scaling a stateful application like Elasticsearch is extremely complicated. A simple QPS or CPU/memory usage metric is not enough to make scaling decisions as there are other considerations such as the amount of data that needs to be shuffled around as the nodes join or leave the cluster, amount of disk space that needs to be kept in reserve to handle a sudden influx of indexing requests, efficiently distributing shards and replicas for availability etc.
There is no support for auto-scaling in ECK. You could, in theory, programmatically modify the nodeSets yourself using your own algorithm but I doubt that would provide a very satisfactory or safe auto-scaling solution. As for HPAs, again, you could create one of your own but it would then compete with the operator and probably destabilise your cluster.
Thanks @charith-elastic for clarifying this. So We can work out HPA with Kibana only as it is a deployment abstract. Please correct me if I am wrong.
Thanks and much appreciated.
Yes, you should be able to use an HPA with Kibana as it is stateless. It is not something that has been extensively tested by us but I don't see a reason why it wouldn't work.
Perfect ! Thanks and much appreciated !
Maybe I am being too naive but what if it is a pure coordinator nodeset? What harm could an HPA setup do?