Elasticsearch start failed after the reboot the server

Hi,
I couldn't start the elasticsearch service after reboot the server. How can i fix it?
I have 3 Mater, 2 Hot, 2 Warm, 2 Cold nodes (Totaly 9 nodes)
Elasticsearch version: 7.7.0
OS: CentOS Linux release 8.1.1911 (Core)

Error Messages;

journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Aug 14 00:40:11 lon-elk-mastr-1 systemd-entrypoint[15145]: Failed to read keystore password on console
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support

Logs;

[2021-08-13T23:42:14,079][ERROR][i.n.u.c.D.rejectedExecution] [lon-elk-mastr-1] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) [netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) [netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109) [netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1012) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:794) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:43) [transport-netty4-client-7.7.0.jar:7.7.0]
        at org.elasticsearch.http.DefaultRestChannel.sendResponse(DefaultRestChannel.java:138) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:505) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:79) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.transport.TransportService$8.run(TransportService.java:1001) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.7.0.jar:7.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-08-13T23:42:14,083][INFO ][o.e.x.m.p.NativeController] [lon-elk-mastr-1] Native controller process has stopped - no new native processes can be started
[2021-08-13T23:42:14,112][INFO ][o.e.n.Node               ] [lon-elk-mastr-1] stopped
[2021-08-13T23:42:14,112][INFO ][o.e.n.Node               ] [lon-elk-mastr-1] closing ...
[2021-08-13T23:42:14,155][INFO ][o.e.n.Node               ] [lon-elk-mastr-1] closed

This is my config file;

cat elasticsearch.yml 

# ---------------------------------- Cluster -----------------------------------
cluster.name: production

# ------------------------------------ Node ------------------------------------
node.name: ${HOSTNAME}
node.master: true
node.data: false
node.ingest: false

# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch/data
path.logs: /var/lib/elasticsearch/logs
path.repo: /mnt/data/es_backup

# ---------------------------------- Network -----------------------------------
http.port: 9200
transport.tcp.port: 9300
network.host: XX.XXX.X.XXX

# --------------------------------- Discovery ----------------------------------

discovery.seed_hosts: ["XX.XXX.X.XXX", "XX.XXX.X.XXX", "XX.XXX.X.XXX"]
cluster.initial_master_nodes: ["XX.XXX.X.XXX", "XX.XXX.X.XXX", "XX.XXX.X.XXX"]


# ---------------------------------- X-Pack -----------------------------------

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates-na.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates-na.p12

Check in the prior startup to see if you had this error, if not, this is probably the cause. Any changes to the server since the last good startup?

I am not aware of changes but I know VM was rebooted. So how can we solve this problem. isn't there another way to fix this ? I disabled the xpack but it didn't work :frowning: Should I reinstall the elasticsearch ? This is production, please help

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.