I am running in my cluster 3 master nodes, two data nodes and three client nodes.
I am facing an issue that we are not able to create new indexes in Elastic.
I can see lot's of exceptions in the elastic search.
It start with:
*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:27.176Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=5, optionalJoin=Optional[Join{term=6, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:447) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:526) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:376) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) [elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:147) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:86) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:801) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*
Then the exceptions ended with:
*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.688Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=8, optionalJoin=Optional[Join{term=10, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-1][192.168.66.228:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 10 does not match current term 11
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:225) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1006) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:525) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:120) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:379) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) ~[elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) ~[?:?]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:267) ~[?:?]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.989Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=10, optionalJoin=Optional[Join{term=11, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.HashMap$Values.forEach(HashMap.java:976) [?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:614) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:250) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*
Any idea what's wrong?