Loss power now my cluster will not connect

So i lost power in the building. My servers came back up but... now my nodes dont communicate. Any help would be appreciated


[ERROR][o.e.b.Elasticsearch      ] [herzai] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch. For more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.17/bootstrap-checks.html]
bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.17/bootstrap-checks-discovery-configuration.html]
ERROR: Elasticsearch did not exit normally - check the logs at /home/dwight-admin/Downloads/elasticsearch-8.17.0/logs/my-application.log

My elasticsearch.yml looks like this

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: herzai
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: herzai
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
transport.host: 0.0.0.0
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["herzai", "zelek","gareb"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["herzai"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 27-12-2024 21:30:17
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------



I referenced both Discovery configuration check | Elasticsearch Guide [8.17] | Elastic and Bootstrap Checks | Elasticsearch Guide [8.17] | Elastic

Were these the 3 nodes of a fully-formed-cluster before you lost power?

yes, they were a fully formed cluster

then comment out that line , on all 3 nodes, and restart all 3 nodes. Your 3 node cluster should re-form.

The error specifically says:

//
At least one of

  • discovery.seed_hosts
  • discovery.seed_providers
  • cluster.initial_master_nodes

must be configured.
//

Now, looking at the documentation

After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration and never set it again for this cluster

So -- you should not use cluster.initial_master_nodes

You could use discovery.seed_providers and point to a file, but in this case its simpler to just use the hosts list you already have in discovery.seed_hosts

1 Like

Thank you the other nodes are up and communicating my final node keeps reporting

[herzai] fatal exception while booting Elasticsearchorg.elasticsearch.http.BindHttpException: Failed to bind to 192.168.10.60:[9200-9300]
	at org.elasticsearch.server@8.17.0/org.elasticsearch.http.AbstractHttpServerTransport.bindAddress(AbstractHttpServerTransport.java:227)
	at org.elasticsearch.server@8.17.0/org.elasticsearch.http.AbstractHttpServerTransport.bindServer(AbstractHttpServerTransport.java:194)
	at org.elasticsearch.transport.netty4@8.17.0/org.elasticsearch.http.netty4.Netty4HttpServerTransport.doStart(Netty4HttpServerTransport.java:231)
	at org.elasticsearch.server@8.17.0/org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:51)
	at org.elasticsearch.server@8.17.0/org.elasticsearch.node.Node.start(Node.java:426)

See logs for more details.

So I dont know how (DHCP)but the ip address was changed to ---.---.--.61.

Are you asking a question or just updating?

It's certainly easier to troubleshoot/manage if all your hosts agree on the hostname<->IP address resolution. Check your /etc/hosts on all. Stuff like network.host: herzai needs herzai to map to a valid IP address for that specific host.

And for future, at least use DHCP reservations next time, or fixed IP addresses. DHCP leases have an expiry, often 1d or 3d or 7d.