Hi,
I couldn't start the elasticsearch service after reboot the server. How can i fix it?
I have 3 Mater, 2 Hot, 2 Warm, 2 Cold nodes (Totaly 9 nodes)
Elasticsearch version: 7.7.0
OS: CentOS Linux release 8.1.1911 (Core)
Error Messages;
journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Aug 14 00:40:11 lon-elk-mastr-1 systemd-entrypoint[15145]: Failed to read keystore password on console
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 14 00:40:11 lon-elk-mastr-1 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
Logs;
[2021-08-13T23:42:14,079][ERROR][i.n.u.c.D.rejectedExecution] [lon-elk-mastr-1] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1012) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:794) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:43) [transport-netty4-client-7.7.0.jar:7.7.0]
at org.elasticsearch.http.DefaultRestChannel.sendResponse(DefaultRestChannel.java:138) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:505) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:79) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.transport.TransportService$8.run(TransportService.java:1001) [elasticsearch-7.7.0.jar:7.7.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.7.0.jar:7.7.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-08-13T23:42:14,083][INFO ][o.e.x.m.p.NativeController] [lon-elk-mastr-1] Native controller process has stopped - no new native processes can be started
[2021-08-13T23:42:14,112][INFO ][o.e.n.Node ] [lon-elk-mastr-1] stopped
[2021-08-13T23:42:14,112][INFO ][o.e.n.Node ] [lon-elk-mastr-1] closing ...
[2021-08-13T23:42:14,155][INFO ][o.e.n.Node ] [lon-elk-mastr-1] closed
This is my config file;
cat elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
cluster.name: production
# ------------------------------------ Node ------------------------------------
node.name: ${HOSTNAME}
node.master: true
node.data: false
node.ingest: false
# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch/data
path.logs: /var/lib/elasticsearch/logs
path.repo: /mnt/data/es_backup
# ---------------------------------- Network -----------------------------------
http.port: 9200
transport.tcp.port: 9300
network.host: XX.XXX.X.XXX
# --------------------------------- Discovery ----------------------------------
discovery.seed_hosts: ["XX.XXX.X.XXX", "XX.XXX.X.XXX", "XX.XXX.X.XXX"]
cluster.initial_master_nodes: ["XX.XXX.X.XXX", "XX.XXX.X.XXX", "XX.XXX.X.XXX"]
# ---------------------------------- X-Pack -----------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates-na.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates-na.p12