Unable to start Elasticsearch

[Ifty]

Good Day!

I am trying to setup Elasticsearch in a 3 node cluster, I am getting below error while I am trying to start Elasticsearch,

//

[root@HRC-ESDC-GLES11 log]# systemctl status elasticsearch.service
â elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-07-27 00:17:50 +04; 13min ago
     Docs: http://www.elastic.co
  Process: 7487 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 7487 (code=exited, status=1/FAILURE)

Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 27 00:17:50 HRC-ESDC-GLES11 elasticsearch[7487]: Refer to the log for complete error details.
Jul 27 00:17:50 HRC-ESDC-GLES11 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 27 00:17:50 HRC-ESDC-GLES11 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 27 00:17:50 HRC-ESDC-GLES11 systemd[1]: elasticsearch.service failed.
[root@HRC-ESDC-GLES11 log]#

//

Elasticsearch Logs

[root@HRC-ESDC-GLES11 log]# cat gc.log.0.current
Java HotSpot(TM) 64-Bit Server VM (25.261-b12) for linux-amd64 JRE (1.8.0_261-b12), built on Jun 17 2020 23:41:40 by "java_re" with gcc 7.3.0
Memory: 4k page, physical 65789784k(62189824k free), swap 2097148k(2097148k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=/app1/elasticsearch/log/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app1/elasticsearch/data -XX:InitialHeapSize=34359738368 -XX:MaxHeapSize=34359738368 -XX:MaxNewSize=2006556672 -XX:MaxTenuringThreshold=6 -XX:NewSize=2006556672 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=4013113344 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2022-07-27T00:17:49.630+0400: 5.806: Total time for which application threads were stopped: 0.0002304 seconds, Stopping threads took: 0.0000818 seconds
2022-07-27T00:17:49.647+0400: 5.822: Total time for which application threads were stopped: 0.0002555 seconds, Stopping threads took: 0.0000846 seconds
2022-07-27T00:17:49.706+0400: 5.881: Total time for which application threads were stopped: 0.0003254 seconds, Stopping threads took: 0.0001477 seconds
2022-07-27T00:17:50.024+0400: 6.200: Total time for which application threads were stopped: 0.0004883 seconds, Stopping threads took: 0.0002528 seconds
2022-07-27T00:17:50.025+0400: 6.201: Total time for which application threads were stopped: 0.0001288 seconds, Stopping threads took: 0.0000268 seconds
2022-07-27T00:17:50.043+0400: 6.218: Total time for which application threads were stopped: 0.0003905 seconds, Stopping threads took: 0.0001240 seconds
2022-07-27T00:17:50.066+0400: 6.242: Total time for which application threads were stopped: 0.0003652 seconds, Stopping threads took: 0.0001186 seconds
2022-07-27T00:17:50.136+0400: 6.312: Total time for which application threads were stopped: 0.0003940 seconds, Stopping threads took: 0.0000847 seconds
2022-07-27T00:17:50.183+0400: 6.359: Total time for which application threads were stopped: 0.0004557 seconds, Stopping threads took: 0.0001200 seconds
2022-07-27T00:17:50.389+0400: 6.564: Total time for which application threads were stopped: 0.0004977 seconds, Stopping threads took: 0.0001111 seconds
2022-07-27T00:17:50.550+0400: 6.726: Total time for which application threads were stopped: 0.0008222 seconds, Stopping threads took: 0.0004422 seconds
Heap
 par new generation   total 1763584K, used 313536K [0x00007f3a68000000, 0x00007f3adf990000, 0x00007f3adf990000)
  eden space 1567680K,  20% used [0x00007f3a68000000, 0x00007f3a7b230330, 0x00007f3ac7af0000)
  from space 195904K,   0% used [0x00007f3ac7af0000, 0x00007f3ac7af0000, 0x00007f3ad3a40000)
  to   space 195904K,   0% used [0x00007f3ad3a40000, 0x00007f3ad3a40000, 0x00007f3adf990000)
 concurrent mark-sweep generation total 31594944K, used 0K [0x00007f3adf990000, 0x00007f4268000000, 0x00007f4268000000)
 Metaspace       used 15752K, capacity 16640K, committed 16896K, reserved 18432K

Thank You,

Ifty!

[Ugo_Sangiorgi]

Can you share your elasticsearch.yml config?

[Ifty]

Thanks for your response, Here is ES .yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog-siem
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /app1/elasticsearch/data
#
# Path to log files:
#
path.logs: /app1/elasticsearch/log
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true

#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: _eth0_
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["10.183.128.236", "10.183.128.237", "10.183.128.238"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true

[Ugo_Sangiorgi]

@Ifty

bootstrap.memory_lock: true

Is the memory locked?
steps to fix it: Elasticsearch - Locking Memory for Production – ugosan – notes

[Ifty]

Thanks Mate it worked