I've been running a 3 node cluster of Elasticsearch
6.8.0 for over a year now. Since it is up and running it only runs with 2 nodes, and every time I join the third node it stops working.
In this case the nodes running are
node-3 and when I start
node-2 the cluster starts reallocating shards, but in a certain moment the node stops working. The same behavior was observed in repeated times previously but with
node-3 joining the cluster. Since
node-1 is exposed to the services that interact with ES I can't try the same, but I suspect it would happen also.
Here are node details and config files:
_cat/nodes 10.92.112.138 55 99 3 0.08 0.11 0.13 mdi * node-1 10.92.112.140 71 99 1 0.09 0.06 0.06 mdi - node-3
cluster.name: wilab-prod node.name: node-2 path.data: /elasticsearch_1/data path.logs: /elasticsearch_1/log network.host: elasticwilab02.client.domain discovery.zen.ping.unicast.hosts: ["elasticwilab01.client.domain", "elasticwilab02.client.domain", "elasticwilab03.client.domain"] discovery.zen.minimum_master_nodes: 2 xpack.security.enabled: true xpack.security.http.ssl.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /opt/elasticsearch/elasticsearch-6.8.0/config/elasticsearch-certificates.p12 xpack.security.transport.ssl.truststore.path: /opt/elasticsearch/elasticsearch-6.8.0/config/elasticsearch-certificates.p12 xpack.security.http.ssl.keystore.path: /opt/elasticsearch/elasticsearch-6.8.0/config/elasticsearch-certificates.p12 xpack.security.http.ssl.truststore.path: /opt/elasticsearch/elasticsearch-6.8.0/config/elasticsearch-certificates.p12 xpack.security.http.ssl.client_authentication: optional
The config file is identical for the three nodes, parameters such as node name, host and paths vary between each other.
Does anyone know what might be happening for the third node that joins the cluster to shut down?