Elasticsearch master not discoverable after ansible deployment

[pkaramol (Pantelis Karamolegkos)]

I have a 1 data - 1 master es cluster.
On my master01:

==> /opt/elasticsearch/logs/master01-elastic.dev.my-local-domain-master01-elastic/esa-local-stg-cluster.log <==
[2019-02-08T11:06:21,267][INFO ][o.e.n.Node               ] [master01-elastic] initialized
[2019-02-08T11:06:21,267][INFO ][o.e.n.Node               ] [master01-elastic] starting ...
[2019-02-08T11:06:21,460][INFO ][o.e.t.TransportService   ] [master01-elastic] publish_address {10.18.0.13:9300}, bound_addresses {10.18.0.13:9300}
[2019-02-08T11:06:21,478][INFO ][o.e.b.BootstrapChecks    ] [master01-elastic] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-08T11:06:24,543][INFO ][o.e.c.s.MasterService    ] [master01-elastic] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {master01-elastic}{10kX4tQMTzS0O8AQYvieZw}{GH9oflu7QZuJB_U7sPJDlg}{10.18.0.13}{10.18.0.13:9300}{xpack.installed=true}
[2019-02-08T11:06:24,550][INFO ][o.e.c.s.ClusterApplierService] [master01-elastic] new_master {master01-elastic}{10kX4tQMTzS0O8AQYvieZw}{GH9oflu7QZuJB_U7sPJDlg}{10.18.0.13}{10.18.0.13:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {master01-elastic}{10kX4tQMTzS0O8AQYvieZw}{GH9oflu7QZuJB_U7sPJDlg}{10.18.0.13}{10.18.0.13:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2019-02-08T11:06:24,575][INFO ][o.e.h.n.Netty4HttpServerTransport] [master01-elastic] publish_address {10.18.0.13:9200}, bound_addresses {10.18.0.13:9200}
[2019-02-08T11:06:24,575][INFO ][o.e.n.Node               ] [master01-elastic] started
[2019-02-08T11:06:24,614][INFO ][o.e.l.LicenseService     ] [master01-elastic] license [c2004733-fa30-4249-bb07-d5f2238816ad] mode [basic] - valid
[2019-02-08T11:06:24,615][INFO ][o.e.g.GatewayService     ] [master01-elastic] recovered [0] indices into cluster_state

[root@master01-elastic ~]# systemctl status elasticsearch
● master01-elastic_elasticsearch.service - Elasticsearch-master01-elastic
   Loaded: loaded (/usr/lib/systemd/system/master01-elastic_elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-02-08 11:06:12 EST; 2 days ago
     Docs: http://www.elastic.co
 Main PID: 18695 (java)
   CGroup: /system.slice/master01-elastic_elasticsearch.service
           ├─18695 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding...
           └─18805 /usr/share/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller

Feb 08 11:06:12 master01-elastic systemd[1]: Started Elasticsearch-master01-elastic.

[root@master01-elastic ~]# ss -tula | grep -i 9300
[root@master01-elastic ~]# 

This makes master undiscoverable so in my data01

[2019-02-11T02:24:09,882][WARN ][o.e.d.z.ZenDiscovery     ] [data01-elastic] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again

Also on my data01

[root@data01-elastic ~]# cat /etc/elasticsearch/data01-elastic/elasticsearch.yml | grep -i zen
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: 10.18.0.13:9300



[root@data01-elastic ~]# ping 10.18.0.13
PING 10.18.0.13 (10.18.0.13) 56(84) bytes of data.
64 bytes from 10.18.0.13: icmp_seq=1 ttl=64 time=0.171 ms
64 bytes from 10.18.0.13: icmp_seq=2 ttl=64 time=0.147 ms

How can I further troubleshoot this?

The cluster was deployed using these ansible scripts:

with this configuration for the master:

  - hosts: masters

    tasks:

    - name: Elasticsearch Master Configuration
      import_role:
          name: elastic.elasticsearch
      vars:
        es_instance_name: "{{ ansible_hostname }}"
        es_data_dirs:
          - "{{ data_dir }}"
        es_log_dir: "/opt/elasticsearch/logs"
        es_config:
          node.name: "{{ ansible_hostname }}"
          cluster.name: "{{ cluster_name }}"
          discovery.zen.ping.unicast.hosts: "{% for host in groups['masters'] -%}{{ hostvars[host]['ansible_ens33']['ipv4']['address'] }}:9300{% if not loop.last %},{% endif %}{%- endfor %}"
          http.port: 9200
          transport.tcp.port: 9300
          node.data: false
          node.master: true
          bootstrap.memory_lock: true
          network.host: '{{ ansible_facts["ens33"]["ipv4"]["address"] }}'
          discovery.zen.minimum_master_nodes: 1
        es_xpack_features: []
        es_scripts: false
        es_templates: false
        es_version_lock: true
        es_heap_size: 2g
        es_api_port: 9200

and this for the data

  - hosts: data

    tasks:

    - name: Elasticsearch Data Configuration
      import_role:
          name: elastic.elasticsearch
      vars:
        es_instance_name: "{{ ansible_hostname }}"
        es_data_dirs:
          - "{{ data_dir }}"
        es_log_dir: "/opt/elasticsearch/logs"
        es_config:
          node.name: "{{ ansible_hostname }}"
          cluster.name: "{{ cluster_name }}"
          discovery.zen.ping.unicast.hosts: "{% for host in groups['masters'] -%}{{ hostvars[host]['ansible_ens33']['ipv4']['address'] }}:9300{% if not loop.last %},{% endif %}{%- endfor %}"
          http.port: 9200
          transport.tcp.port: 9300
          node.data: true
          node.master: false
          bootstrap.memory_lock: true
          network.host: '{{ ansible_facts["ens33"]["ipv4"]["address"] }}'
          discovery.zen.minimum_master_nodes: 1
        es_xpack_features: []
        es_scripts: false
        es_templates: false
        es_version_lock: true
        es_heap_size: 6g
        es_api_port: 9200

[pkaramol (Pantelis Karamolegkos)]

The 2 VMs I was trying to establish communication among were Centos7 which has firewalld enabled by default.

Disabling and stopping the service solved the issue.