# Configuration issue

**URL:** <https://discuss.elastic.co/t/configuration-issue/274839>\
**Category:** Elasticsearch\
**Created:** [June 3, 2021, 9:14am UTC](https://discuss.elastic.co/t/configuration-issue/274839 "2021-06-03T09:14:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [June 3, 2021, 9:14am UTC](https://discuss.elastic.co/t/configuration-issue/274839/1 "2021-06-03T09:14:29Z")

</div>

I am running in my cluster 3 master nodes, two data nodes and three client nodes.  
I am facing an issue that we are not able to create new indexes in Elastic.  
I can see lot's of exceptions in the elastic search.

It start with:  
\*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:27.176Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=5, optionalJoin=Optional[Join{term=6, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}  
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}  
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:447) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:526) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:376) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) [elasticsearch-7.8.0.jar:7.8.0]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:147) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:86) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:801) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
at java.lang.Thread.run(Thread.java:834) [?:?]  
\*

Then the exceptions ended with:

\*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.688Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=8, optionalJoin=Optional[Join{term=10, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-1}{Q75wEsF-RA2wal7I7sCm9A}{0QuK6FXZTkGkxblsW7x-4A}{192.168.66.228}{192.168.66.228:9300}{m}}]}"}}  
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-1][192.168.66.228:9300][internal:cluster/coordination/join]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 10 does not match current term 11  
at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:225) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1006) ~[elasticsearch-7.8.0.jar:7.8.0]  
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]  
at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:525) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:489) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:120) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:379) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:363) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:476) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:129) ~[elasticsearch-7.8.0.jar:7.8.0]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) ~[?:?]  
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:267) ~[?:?]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) ~[?:?]  
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) ~[?:?]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.0.jar:7.8.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]  
at java.lang.Thread.run(Thread.java:834) [?:?]  
{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:53:28.989Z","logger":"o.e.c.c.JoinHelper","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"failed to join {po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m} with JoinRequest{sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, minimumTerm=10, optionalJoin=Optional[Join{term=11, lastAcceptedTerm=5, lastAcceptedVersion=217, sourceNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}, targetNode={po-btel-es-master-0}{LGpijZohQ6yuOes3DJnzig}{8u8-8QzMSqOJ8KehQENmqA}{192.168.0.48}{192.168.0.48:9300}{m}}]}"}}  
org.elasticsearch.transport.RemoteTransportException: [po-btel-es-master-0][192.168.0.48:9300][internal:cluster/coordination/join]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower  
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]  
at java.util.HashMap$Values.forEach(HashMap.java:976) [?:?]  
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:472) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:614) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:250) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
at java.lang.Thread.run(Thread.java:834) [?:?]  
\*

Any idea what's wrong?

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [June 3, 2021, 9:16am UTC](https://discuss.elastic.co/t/configuration-issue/274839/2 "2021-06-03T09:16:09Z")

</div>

I can see NullPointerException in the end of the log:

\*{"type":"log","host":"po-btel-es-master-0","level":"INFO","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:54:14.290Z","logger":"o.q.c.QuartzScheduler","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"Scheduler signals/_main_$\_signals/\_main started."}}  
run\_as\_other\_user\_if\_needed /usr/share/elasticsearch/bin/elasticsearch "${es\_opts[@]}"  
{"type":"log","host":"po-btel-es-master-0","level":"WARN","systemid":"2106a117733f42d697284fbc54927928","system":"BELK","time": "2021-06-02T13:57:48.625Z","logger":"o.e.t.OutboundHandler","timezone":"UTC","marker":"[po-btel-es-master-0] ","log":{"message":"send message failed [channel: Netty4TcpChannel{localAddress=/192.168.0.48:9300, remoteAddress=/192.168.66.227:60046}]"}}  
java.lang.NullPointerException: null  
at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:435) ~[elasticsearch-7.8.0.jar:7.8.0]  
at com.floragunn.searchguard.support.ModuleInfo.writeTo(ModuleInfo.java:99) ~[?:?]  
at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeCollection$31(StreamOutput.java:1160) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1179) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1160) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.io.stream.StreamOutput.writeList(StreamOutput.java:1167) ~[elasticsearch-7.8.0.jar:7.8.0]  
at com.floragunn.searchguard.action.licenseinfo.LicenseInfoNodeResponse.writeTo(LicenseInfoNodeResponse.java:66) ~[?:?]  
at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:162) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:148) ~[elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:195) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:129) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:124) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:104) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:63) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.8.0.jar:7.8.0]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:164) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:267) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:136) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at com.floragunn.searchguard.SearchGuardPlugin$7.lambda$interceptHandler$0(SearchGuardPlugin.java:683) [search-guard-suite-security-7.8.0-43.0.0-146.jar:7.8.0-43.0.0-146]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
at java.lang.Thread.run(Thread.java:834) [?:?]  
\*

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 7, 2021, 6:01am UTC](https://discuss.elastic.co/t/configuration-issue/274839/3 "2021-06-07T06:01:51Z")

</div>

> [@ssasporta](#):
>
> I can see lot's of exceptions in the Elasticsearch.

What does your elasticsearch.yml look like?

Also you should upgrade to 7.13 and use the free, inbuilt Elasticsearch Security functionality!

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [June 7, 2021, 6:42am UTC](https://discuss.elastic.co/t/configuration-issue/274839/4 "2021-06-07T06:42:29Z")

</div>

Hi Mark,  
Which nodes of the Elasticsearch? Master, Data or Client?  
Thanks  
Sharon

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 7, 2021, 6:44am UTC](https://discuss.elastic.co/t/configuration-issue/274839/5 "2021-06-07T06:44:03Z")

</div>

The stack traces seem related to SearchGuard. I would recommend you reach out to them for support.

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [June 7, 2021, 6:48am UTC](https://discuss.elastic.co/t/configuration-issue/274839/6 "2021-06-07T06:48:19Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> SearchGuard

Thanks Christian. You are right!!! Will update. Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2021, 6:48am UTC](https://discuss.elastic.co/t/configuration-issue/274839/7 "2021-07-05T06:48:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.