ElasticSearch on Kubernetes, one master nod not discovered when xpack.security.transport.ssl.enabled=true

[Uvais_Ibrahim]

I have deployed an elasticsearch in Kubernetes with 3 masters and 3 data nodes. All the pods are running.

NAME                        READY   STATUS    RESTARTS   AGE
elasticsearch-master-v1-0   1/1     Running   0          29m
elasticsearch-master-v1-1   1/1     Running   0          29m
elasticsearch-master-v1-2   1/1     Running   0          29m

Note: Now the replicaset for data-node deployment is set to 0.

Elasticsearch Version: 6.4.0

But when I had checked the node status through elasticsearch API, only two nodes are showing up.

API CALL: /_cat/nodes?v

ip              heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.188.158.230            2          38  27    0.35    0.21     0.15 m         *      elasticsearch-master-v1-1
192.188.119.232            2          38  19    0.55    0.26     0.17 m         -      elasticsearch-master-v1-0

It should show all the three master nodes above

Why the third node is not showing up here? The following are the relevant configuration in elasticsearch.yml file.

discovery:
      zen:
        ping.unicast.hosts: elasticsearch-discovery  
//This is the service I created for internode communication.
        minimum_master_nodes: 2

    xpack.ml.enabled: false
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

I noticed that, when in the above configuration xpack.security.transport.ssl.enabled: false on the same API call it showed all the three master nodes

K8s Service yaml for inter node communication.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch-master
  name: elasticsearch-discovery-master
spec:
  ports:
  - name: transport
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch-master
  type: ClusterIP

This is what the logs says from the third master node which is not discovered,

[2020-08-12T12:38:41,342][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:38:50,616][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:39:15,424][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:39:18,425][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again