Ansible Elasticsearch Hangs on TASK [elastic.elasticsearch : Wait for elasticsearch to startup]

[M_M (Miroslav)]

The elasticsearch initiates successfully on the nodes but somehow ansible is stuck at
TASK [elastic.elasticsearch : Wait for elasticsearch to startup]
My nodes are EC2 Amazon linux 2 and ansible is run from my desktop ansible-2.9.10-1.fc32.noarch.
Ansible config file:

- hosts: ssh-node1
  roles:
    - role: elastic.elasticsearch
  vars:
    es_heap_size: "1g"
    es_data_dirs:
      - "/opt/elasticsearch"  
    es_config:
      node.name: "node-1"
      cluster.name: "ansible-cluster"
      cluster.initial_master_nodes: "172.XXX.XXX.111"
      discovery.seed_hosts: "172.XXX.XXX.111:9300"
      network.host: "_eth0_"  
      node.data: true
      node.master: true
      xpack.security.authc.realms.file.file1.order: 0
      xpack.security.authc.realms.native.native1.order: 1
    es_api_basic_auth_username: elastic
    es_api_basic_auth_password: changeme
    es_enable_http_ssl: true
    es_enable_transport_ssl: true
    es_ssl_keystore: "ssl_certs/my-keystore.p12"
    es_ssl_truststore: "ssl_certs/my-keystore.p12"
    es_ssl_keystore_password: "keystore_password"
    es_ssl_truststore_password: "keystore_password"
    es_validate_certs: no
    
- hosts: ssh-node2
  roles:
    - role: elastic.elasticsearch
  vars:
    es_heap_size: "1g"
    es_data_dirs:
      - "/opt/elasticsearch"  
    es_config:
      node.name: "node-2"
      cluster.name: "ansible-cluster"
      cluster.initial_master_nodes: "172.XXX.XXX.111"
      discovery.seed_hosts: "172.XXX.XXX.111:9300"
      network.host: "_eth0_"  
      node.data: true
      node.master: true
      xpack.security.authc.realms.file.file1.order: 0
      xpack.security.authc.realms.native.native1.order: 1
    es_api_basic_auth_username: elastic
    es_api_basic_auth_password: changeme
    es_enable_http_ssl: true
    es_enable_transport_ssl: true
    es_ssl_keystore: "ssl_certs/my-keystore.p12"
    es_ssl_truststore: "ssl_certs/my-keystore.p12"
    es_ssl_keystore_password: "keystore_password"
    es_ssl_truststore_password: "keystore_password"
    es_validate_certs: no

Output

TASK [elastic.elasticsearch : Wait for elasticsearch to startup] ***********************************************
fatal: [obj-ansible-node1]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for localhost:9200"}

PLAY RECAP *****************************************************************************************************
ssh-node1          : ok=34   changed=12   unreachable=0    failed=1    skipped=84   rescued=0    ignored=0   

After restarting the script it successfully goes trough node-1 tasks and hangs the same way on node-2.
At the end I do have fully working 2-node SSL/TLS secured cluster.

What is wrong with my configuration? Any help is greatly appreciated.

[M_M (Miroslav)]

Found the issue.
As it says in the documentation:
README->Important Note

The role uses es_api_host and es_api_port to communicate with the node for actions only achievable via http e.g. to install templates and to check the NODE IS ACTIVE. These default to "localhost" and 9200 respectively. If the node is deployed to bind on either a different host or port, these must be changed.

This correction fixed the issue.

- hosts: ssh-node1
  ...
  es_api_host: "172.XXX.XXX.111"

- hosts: ssh-node2
  ...
  es_api_host: "172.XXX.XXX.222"

On a separate note for future reference, the SSL/TLS certificates were generated in advance on an existing Elasticsearch node as described in the documentation.

elastic/ansible-elasticsearch/blob/main/docs/ssl-tls-setup.md

[M_M (Miroslav)]

Here is a shorter universal playbook which would work for cluster of any size:

- hosts: all
  roles:
    - role: elastic.elasticsearch
  vars:
    seed_hosts: "[{%for host in groups['es-nodes']%}\"{{hostvars[host].ansible_eth0.ipv4.address}}:9300\"{% if not loop.last %},{% endif %}{% endfor %}]"
    master_nodes: "[{%for host in groups['es-nodes']%}\"{{hostvars[host].ansible_eth0.ipv4.address}}\"{% if not loop.last %},{% endif %}{% endfor %}]"

    es_heap_size: "{{hostvars[inventory_hostname].heap_size}}"
    es_data_dirs:
      - "/opt/elasticsearch"
    es_api_host: "{{ ansible_default_ipv4.address}}"  
    es_config:
      node.name: "{{hostvars[inventory_hostname].node_name}}"
      cluster.name: "{{hostvars[inventory_hostname].cluster_name}}"
      cluster.initial_master_nodes: "{{master_nodes}}"
      discovery.seed_hosts: "{{ seed_hosts }}"
      network.host: "_eth0_"  
      node.data: true
      node.master: true
      xpack.security.authc.realms.file.file1.order: 0
      xpack.security.authc.realms.native.native1.order: 1
    es_api_basic_auth_username: elastic
    es_api_basic_auth_password: changeme
    es_enable_http_ssl: true
    es_enable_transport_ssl: true
    es_ssl_keystore: "ssl_certs/my-keystore.p12"
    es_ssl_truststore: "ssl_certs/my-keystore.p12"
    es_ssl_keystore_password: "elastic"
    es_ssl_truststore_password: "elastic"
    es_validate_certs: no

And the inventory file:

[es-nodes]
ssh-alias-node1 node_name=node-1
ssh-alias-node2 node_name=node-2
ssh-alias-node3 node_name=node-3

[es-nodes:vars]
heap_size=1g
cluster_name=ansible-cluster

ansible-playbook -i inv.conf es-playbook.yml