ALB health check failure while checking Elasticsearch cluster health

[siddharthavempa (siddhartha)]

Hi Team,

I am creating a two node Elasticsearch cluster in AWS using EC2 instances.

I installed Elasticsearch in node-1 using rpm. Then I used the below commands to modify the elasticsearch.yaml file and started elasticsearch service

HOSTNAME=$(hostname | awk -F"." '{print $1}')
IPV4_ADDRESS=$(hostname -I)
sed -i '/#cluster.name/s/#cluster.name: my-application/cluster.name: es-demo/' /etc/elasticsearch/elasticsearch.yml
sed -i "/#node.name/s/#node.name: node-1/node.name: $HOSTNAME/" /etc/elasticsearch/elasticsearch.yml
sed -i "/#network.host/s/#network.host: 192.168.0.1/network.host: $IPV4_ADDRESS/" /etc/elasticsearch/elasticsearch.yml
sed -i '/#http.port/s/#http.port: 9200/http.port: 9200/' /etc/elasticsearch/elasticsearch.yml
sed -i "/cos.dev.nonprod.x1.example.com/s/$HOSTNAME.cos.dev.nonprod.x1.example.com/$HOSTNAME/" /etc/elasticsearch/elasticsearch.yml
sed -i '/#transport.host/s/#transport.host: 0.0.0.0/transport.host: 0.0.0.0/' /etc/elasticsearch/elasticsearch.yml

Then started the elasticsearch service using the command "systemctl start elasticsearch.service"

Using the below command generated enrollment token for new node
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

I am using auto configured security settings here so the installation of elasticsearch created the certificates used for ssl of http and transport layer.

Then I installed Elasticsearch in node-2 using rpm. Then I used the above generated node enrollment token and executed the below command to join node-2 into the cluster.
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token "${NODE_ENROLLMENT_TOKEN}"

Then I used the below commands to modify the elasticsearch.yaml file and started elasticsearch service

HOSTNAME=$(hostname | awk -F"." '{print $1}')
IPV4_ADDRESS=$(hostname -I)
sed -i '/#cluster.name/s/#cluster.name: my-application/cluster.name: es-demo/' /etc/elasticsearch/elasticsearch.yml
sed -i "/#node.name/s/#node.name: node-1/node.name: $HOSTNAME/" /etc/elasticsearch/elasticsearch.yml
sed -i "/#network.host/s/#network.host: 192.168.0.1/network.host: $IPV4_ADDRESS/" /etc/elasticsearch/elasticsearch.yml
sed -i '/#http.port/s/#http.port: 9200/http.port: 9200/' /etc/elasticsearch/elasticsearch.yml

Then started the elasticsearch service using the command "systemctl start elasticsearch.service"

I made a curl request from both the nodes using the below command and it worked well.
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200/_cluster/health?pretty

Then I created an AWS TargetGroup and Application LoadBalancer and coupled both of them with the respective rules.
Then I registered these two nodes into the TargetGroup. The nodes security groups are allowed to accept any traffic from the security group of the Application LoadBalancer security group.
The TargetGroup health check path is set to /_cluster/health. But the health checks are failing. I understood that loadbalancer is not using the --cacert /etc/elasticsearch/certs/http_ca.crt to trust the certificate and also due to lack of authentication details like username and password in the load balancer health check request.

Now I need your assistance on how to address this situation. How can I make my AWS ALB trust the certificate created by Elasticsearch during its installation and also how can I pass the cluster username and password so that Load Balancer health checks can authenticate with the cluster and get 200 response for the health check page. Kindly please help me with this.