Deploy an auto-scaling cluster in AWS Elastic Beanstalk

[gpribi (GABRIEL PRZYBYLSKI)]

Hi, I currently have a single-node production environment in AWS EC2 and having scaling issues, so I wanted to make the transition into a cluster. Since my user demand changes a lot during the day, I thought that a good idea would be to have an Beanstalk environment with a LB and a permanent master node. When the demand grows up, Beanstalk can create slave instances and balance the requests throug all the cluster.

The problem is how to configure this, since the cluster config has static values, such as the network.host, discovery.seed_hosts, cluster.initial_master_nodes,... and the node.name is optional but I don't know which value will take once the instance starts.

I plan to have also a EnterpriseSearch container running in the same server, same as I have now, and I'm using docker compose to build all the containers.

I've searched for this situation and found very old posts refering to deprecated properties (for example GitHub - vladmiller/elasticsearch-beanstalk: ElasticSearch configuration example for AWS Elastic BeansTalk). I guess both ELK and Beanstalk have improved the last 6 years and it would be easier t configure!

Thanks in advance
Gabriel

[DavidTurner]

@gpribi

since the cluster config has static values, such as the network.host, discovery.seed_hosts, cluster.initial_master_nodes

network.host can refer to an interface, e.g. _eth0_, or you can pick a site-local address with _site_.

discovery.seed_hosts can be a DNS name, or you can use a different discovery mechanism like unicast_hosts.txt or EC2 Discovery Plugin | Elasticsearch Plugins and Integrations [7.15] | Elastic.

cluster.initial_master_nodes shouldn't be set once the cluster is formed.

However, scaling a cluster up and down so much tends to be a bad idea, each time you scale-up ES will copy a bunch of data over the network to build replicas, which takes a while and costs money.