The cluster is not working

I can't start the elasticsearch cluster

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: es_cluster
cluster.initial_master_nodes:["node-01", "node-02", "node-03']
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-01
node.roles: [master, data]
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.0.66
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["node-01", "node-02", "node-03"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 20-05-2024 12:40:41
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true
#xpack.security.enabled: false
xpack.security.enrollment.enabled: true
#xpack.security.enrollment.enabled: false


# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

#xpack.security.http.ssl:
#  enabled: false
#  keystore.path: certs/http.p12


# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["node-01k"]
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["node-01"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

$ systemctl start elasticsearch.service

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

$ systemctl status elasticsearch.service

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset:
   Active: failed (Result: exit-code) since Tue 2024-05-21 08:10:23 MSK; 40min ago
     Docs: https://www.elastic.co
  Process: 16504 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/el
 Main PID: 16504 (code=exited, status=1/FAILURE)

May 21 08:10:23 node-01 systemd-entrypoint[16504]:  at [Source: (ByteArrayInputStream
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         at com.fasterxml.jackson.c
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         at com.fasterxml.jackson.c
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         at com.fasterxml.jackson.d
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         at com.fasterxml.jackson.c
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         at org.elasticsearch.xcont
May 21 08:10:23 node-01 systemd-entrypoint[16504]:         ... 10 more
May 21 08:10:23 node-01 systemd[1]: elasticsearch.service: Main process exited, code=
May 21 08:10:23 node-01 systemd[1]: elasticsearch.service: Failed with result 'exit-c
May 21 08:10:23 node-01 systemd[1]: Failed to start Elasticsearch.

tell me please what is the problem?

Hey @ascalonking , the outputs for the service you included in your post are truncated, the stacktrace is not completely visible.

Can you provide them so we can try to diagnose the issue?

Hi @Carlos_D

systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2024-05-21 10:52:26 MSK; 5s ago
     Docs: https://www.elastic.co
  Process: 19536 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 19536 (code=exited, status=1/FAILURE)

May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:307)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:237)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
May 21 10:52:26 node-01 systemd-entrypoint[19536]:         ... 12 more
May 21 10:52:26 node-01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
May 21 10:52:26 node-01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
May 21 10:52:26 node-01 systemd[1]: Failed to start Elasticsearch.

Thank you

Hey @ascalonking :

You have several errors in your YAML config file. Please use a YAML validator like this to double check you have a correct config file.

Line 18 should read:

cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]

And you should remove the duplicated cluster.initial_master_nodes key on line 77.

Hope that helps!

@Carlos_D Thank you very much, your solution helped me. But now I have another problem, my nodes don’t want to form a cluster

[username@node-01 ~]# curl -X GET "http://192.168.0.68:9300/_cluster/health?pretty"
curl: (52) Empty reply from server
[username@node-01 ~]# tail -n 25 /var/log/elasticsearch/es_cluster.log
[2024-05-23T20:59:25,328][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-01] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/192.168.0.66:9300, remoteAddress=/192.168.0.67:42892, profile=default}
[2024-05-23T20:59:39,878][WARN ][o.e.c.c.Coordinator      ] [node-01] This node is a fully-formed single-node cluster with cluster UUID [KPxh***********97iNg], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[node-01, node-02, node-03]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

The firewall is disabled on all nodes

I solved the problem with UUID with the following command

# rm -rf /var/lib/elasticsearch/*

# systemctl restart elasticsearch.service

But now I have another problem, my nodes can't see other nodes

][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-01] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/192.168.0.66:9300, remoteAddress=/192.168.0.68:55452, profile=default}

I'm glad you're solving your issues @ascalonking !

You should check how to configure SSL or disable TLS in case you're doing some tests (not recommended for production!)

My cluster started working, thank you! I'm currently testing the ELK stack.

1 Like