Configuration issue

I am running in my cluster 3 master nodes, two data nodes and three client nodes.
I am facing an issue that we are not able to create new indexes in Elastic.
I can see lot's of exceptions in the elastic search.

It start with:
*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:27.176Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=5, optionalJoin=Optional[Join{term=6, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:447) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:526) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:376) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) [elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:147) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:86) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:801) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*

Then the exceptions ended with:

*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.688Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=8, optionalJoin=Optional[Join{term=10, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-1][192.168.66.228:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 10 does not match current term 11
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:225) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1006) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:525) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:120) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:379) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) ~[elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) ~[?:?]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:267) ~[?:?]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.989Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=10, optionalJoin=Optional[Join{term=11, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.HashMap$Values.forEach(HashMap.java:976) [?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:614) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:250) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*

Any idea what's wrong?

I can see NullPointerException in the end of the log:

*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:54:14.290Z","logger":"o.q.c.QuartzScheduler","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"Scheduler signals/main$_signals/_main started."}}
run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}"
{"type":"log","host":"po-btel-es-master-0","level":"WARN","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:57:48.625Z","logger":"o.e.t.OutboundHandler","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"send message failed [channel: Netty4TcpChannel{localAddress=/192.168.0.48:9300, remoteAddress=/192.168.66.227:60046}]"}}
java.lang.NullPointerException: null
at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:435) ~[elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.support.ModuleInfo.writeTo(ModuleInfo.java:99) ~[?:?]
at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeCollection$31(StreamOutput.java:1160) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1179) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1160) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.io.stream.StreamOutput.writeList(StreamOutput.java:1167) ~[elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.action.licenseinfo.LicenseInfoNodeResponse.writeTo(LicenseInfoNodeResponse.java:66) ~[?:?]
at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:162) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:148) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:195) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:129) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:124) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:104) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.8.0.jar:7.8.0]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:267) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
*

What does your elasticsearch.yml look like?

Also you should upgrade to 7.13 and use the free, inbuilt Elasticsearch Security functionality!

Hi Mark,
Which nodes of the Elasticsearch? Master, Data or Client?
Thanks
Sharon

The stack traces seem related to SearchGuard. I would recommend you reach out to them for support.

Thanks Christian. You are right!!! Will update. Thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.