# Master is not joiing cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2017, 7:48pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387 "2017-10-01T19:48:12Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ghanshyam](https://avatars.discourse-cdn.com/v4/letter/g/ba8739/32.png) [@ghanshyam](https://discuss.elastic.co/u/ghanshyam)\ **Post date:** [October 1, 2017, 7:48pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/1 "2017-10-01T19:48:12Z")
I have 3 master, 2 is up and but third one is not joining. Node1: [c810wem.int.thomsonreuters.com](http://c810wem.int.thomsonreuters.com) [node.name](http://node.name): ${HOSTNAME}-master-${ES\_HTTP\_PORT} http.cors.enabled: true tr.security.transport.ipfilter: "+n:c628agv.int.thomsonreuters.com, +n:c219usb.int.thomsonreuters.com, +n:c128rrs.int.thomsonreuters.com, +n:c810wem.int.thomsonreuters.com, +n:c014typ.int.thomsonreuters.com, +n:localhost, -n:_" path.plugins: ${INSTANCE\_BASE}/plugins cluster.routing.allocation.same\_shard.host: true bootstrap.mlockall: true path.data: /data/es [cluster.name](http://cluster.name): es-WLN-TypeAhead-Lab tr.security.authorization.cache.refresh\_minutes: 5 http.port: ${ES\_HTTP\_PORT} node.master: true tr.security.enabled: true discovery.zen.minimum\_master\_nodes: 3 tr.security.authentication.basic.enabled: true tr.security.authentication.safe.enabled: true http.cors.allow-credentials: true path.logs: /log http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Authorization" +tr.security.authentication.safe.app\_key: tyIBqgOBJbiBpbpCbL5/1K5XHQVsJk9fMZ6jb5PlEPg=;EYHm7MewFCWbw8KqI3B2Nw== discovery.zen.ping.unicast.hosts: ["[c628agv.int.thomsonreuters.com:9300](http://c628agv.int.thomsonreuters.com:9300)", "[c014typ.int.thomsonreuters.com:9301](http://c014typ.int.thomsonreuters.com:9301)", "[c810wem.int.thomsonreuters.com:9302](http://c810wem.int.thomsonreuters.com:9302)"] security.manager.enabled: false node.appserver\_instance: ${APPSERVER\_INSTANCE} network.host: 0 node.data: false http.cors.allow-origin: "_" tr.version: ${RELEASE} Node2: [c014typ.int.thomsonreuters.com](http://c014typ.int.thomsonreuters.com) http.cors.enabled: true tr.security.transport.ipfilter: "+n:c628agv.int.thomsonreuters.com, +n:c219usb.int.thomsonreuters.com, +n:c128rrs.int.thomsonreuters.com, +n:c810wem.int.thomsonreuters.com, +n:c014typ.int.thomsonreuters.com, +n:localhost, -n:_" path.plugins: ${INSTANCE\_BASE}/plugins cluster.routing.allocation.same\_shard.host: true bootstrap.mlockall: true path.data: /data/es [cluster.name](http://cluster.name): es-WLN-TypeAhead-Lab tr.security.authorization.cache.refresh\_minutes: 5 http.port: ${ES\_HTTP\_PORT} node.master: true tr.security.enabled: true discovery.zen.minimum\_master\_nodes: 3 tr.security.authentication.basic.enabled: true tr.security.authentication.safe.enabled: true http.cors.allow-credentials: true path.logs: /log http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Authorization" discovery.zen.ping.unicast.hosts: ["[c628agv.int.thomsonreuters.com:9300](http://c628agv.int.thomsonreuters.com:9300)", "[c014typ.int.thomsonreuters.com:9300](http://c014typ.int.thomsonreuters.com:9300)", "[c014typ.int.thomsonreuters.com:9301](http://c014typ.int.thomsonreuters.com:9301)"] security.manager.enabled: false node.appserver\_instance: ${APPSERVER\_INSTANCE} network.host: 0 node.data: false http.cors.allow-origin: "_" tr.version: ${RELEASE} transport.tcp.port: ${ES\_TRANSPORT\_PORT} discovery.zen.ping.multicast.enabled: false Node 3 which is not joing; [c628agv.int.thomsonreuters.com](http://c628agv.int.thomsonreuters.com) http.cors.enabled: true tr.security.transport.ipfilter: "+n:c628agv.int.thomsonreuters.com, +n:c219usb.int.thomsonreuters.com, +n:c128rrs.int.thomsonreuters.com, +n:c810wem.int.thomsonreuters.com, +n:c014typ.int.thomsonreuters.com, +n:localhost, -n:_" path.plugins: ${INSTANCE\_BASE}/plugins cluster.routing.allocation.same\_shard.host: true bootstrap.mlockall: true path.data: /data/es [cluster.name](http://cluster.name): es-WLN-TypeAhead-Lab tr.security.authorization.cache.refresh\_minutes: 5 http.port: ${ES\_HTTP\_PORT} node.master: true tr.security.enabled: true discovery.zen.minimum\_master\_nodes: 3 tr.security.authentication.basic.enabled: true tr.security.authentication.safe.enabled: true http.cors.allow-credentials: true path.logs: /log http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Authorization" discovery.zen.ping.unicast.hosts: ["[c628agv.int.thomsonreuters.com:9303](http://c628agv.int.thomsonreuters.com:9303)", "[c014typ.int.thomsonreuters.com:9301](http://c014typ.int.thomsonreuters.com:9301)", "[c810wem.int.thomsonreuters.com:9302](http://c810wem.int.thomsonreuters.com:9302)"] security.manager.enabled: false node.appserver\_instance: ${APPSERVER\_INSTANCE} node.data: false http.cors.allow-origin: "_" tr.version: ${RELEASE} transport.tcp.port: ${ES\_TRANSPORT\_PORT} discovery.zen.ping.multicast.enabled: false and the error is: ``` --> ping_response{node [{c014typ-client-9201}{zrNZVNhDQbeb2Q5WgOx-Fg}{10.204.101.87}{10.204.101.87:9301}{data=false, appserver_instance=es-WLN-TypeAhead-Lab_2.4.2.2_9401, master=true}], id[10848], master [{c810wem-master-9200}{GtZClXtTR-2iOz58_VFC7A}{10.204.101.86}{10.204.101.86:9300}{data=false, appserver_instance=es-WLN-TypeAhead-Lab_2.4.2.2_9400, master=true}], hasJoinedOnce [true], cluster_name[es-WLN-TypeAhead-Lab]} ``` [2017-10-01T18:57:26.295Z][DEBUG][discovery.zen.publish][c628agv-master-9200] received full cluster state version 82 with size 168267 [2017-10-01T18:57:26.297Z][DEBUG][discovery.zen.fd][c628agv-master-9200] [master] restarting fault detection against master [{c810wem-master-9200}{GtZClXtTR-2iOz58\_VFC7A}{10.204.101.86}{10.204.101.86:9300}{data=false, appserver\_instance=es-WLN-TypeAhead-Lab\_2.4.2.2\_9400, master=true}], reason [new cluster state received and we are monitoring the wrong master [null]] o be in VALID state... [2017-10-01T19:16:12.292Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:13.295Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:14.298Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:15.301Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:16.304Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:17.308Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... [2017-10-01T19:16:18.311Z][INFO][com.trgr.elasticsearch.plugin.security.SecurityBootstrapService][c628agv-master-9200] Waiting for Cluster to be in VALID state... ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 1, 2017, 9:15pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/2 "2017-10-01T21:15:35Z")
What are all the settings that begin with `tr.`? ---
**Author:** ![ghanshyam](https://avatars.discourse-cdn.com/v4/letter/g/ba8739/32.png) [@ghanshyam](https://discuss.elastic.co/u/ghanshyam)\ **Post date:** [October 2, 2017, 2:22pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/3 "2017-10-02T14:22:03Z")
these are our customized security setting instead of Shild ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 2, 2017, 2:31pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/4 "2017-10-02T14:31:35Z")
If you have a completely custom security implementation in place that could impact the traffic within the cluster, I suspect it may be difficult for anyone here to be able to help out. On another probably unrelated note, you should really have `minimum_master_nodes` set to `2` instead of `3` assuming you have 3 master eligible nodes. ---
**Author:** ![ghanshyam](https://avatars.discourse-cdn.com/v4/letter/g/ba8739/32.png) [@ghanshyam](https://discuss.elastic.co/u/ghanshyam)\ **Post date:** [October 2, 2017, 2:43pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/5 "2017-10-02T14:43:57Z")
ok, I can try minimum\_master\_nodes set to 2 and see the status. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 2, 2017, 2:52pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/6 "2017-10-02T14:52:00Z")
It should allow the cluster to start up, but will probably not help the third master node join. ---
**Author:** ![ghanshyam](https://avatars.discourse-cdn.com/v4/letter/g/ba8739/32.png) [@ghanshyam](https://discuss.elastic.co/u/ghanshyam)\ **Post date:** [October 2, 2017, 4:25pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/7 "2017-10-02T16:25:01Z")
cluster is up now but now all access has gone and now we are getting this error: HTTP status code: 403: authentication failed because the cluster is still recovering the security index ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 2, 2017, 4:26pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/8 "2017-10-02T16:26:51Z")
That I can not help you with as I know nothing about your security arrangement. ---
**Author:** ![ghanshyam](https://avatars.discourse-cdn.com/v4/letter/g/ba8739/32.png) [@ghanshyam](https://discuss.elastic.co/u/ghanshyam)\ **Post date:** [October 2, 2017, 4:53pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/9 "2017-10-02T16:53:19Z")
thank you Christian!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 30, 2017, 4:53pm UTC](https://discuss.elastic.co/t/master-is-not-joiing-cluster/102387/10 "2017-10-30T16:53:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.