# Master election issue? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 24, 2012, 6:57pm UTC](https://discuss.elastic.co/t/master-election-issue/6802 "2012-02-24T18:57:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [February 24, 2012, 6:57pm UTC](https://discuss.elastic.co/t/master-election-issue/6802/1 "2012-02-24T18:57:38Z")
After full cluster restart, my master is unable to elect itself, here are some logs: [2012-02-24 10:55:21,549][INFO][discovery.zen] [img700] failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] [inet[/208.94.2.146:9300]]{master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ 208.94.2.146:9300]][discovery/zen/join]; org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ 208.94.2.146:9300]]{master=true}]] [2012-02-24 10:55:24,561][INFO][discovery.zen] [img700] failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] [inet[/208.94.2.146:9300]]{master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ 208.94.2.146:9300]][discovery/zen/join]; org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ 208.94.2.146:9300]]{master=true}]] [2012-02-24 10:55:27,568][INFO][discovery.zen] [img700] failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] [inet[/208.94.2.146:9300]]{master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ 208.94.2.146:9300]][discovery/zen/join]; org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ 208.94.2.146:9300]]{master=true}]] [2012-02-24 10:55:30,778][INFO][discovery.zen] [img700] failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] [inet[/208.94.2.146:9300]]{master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ 208.94.2.146:9300]][discovery/zen/join]; org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ 208.94.2.146:9300]]{master=true}]] This has been going on for 10 minutes or so... Any ideas what might be wrong? -Jack ---
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [February 26, 2012, 3:56am UTC](https://discuss.elastic.co/t/master-election-issue/6802/2 "2012-02-26T03:56:52Z")
I was able to resolve this by shutting down all nodes in the cluster, and starting up masters first. -Jack On Feb 24, 10:57 am, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > After full cluster restart, my master is unable to elect itself, here > are some logs: > > [2012-02-24 10:55:21,549][INFO][discovery.zen] [img700] > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > [inet[/208.94.2.146:9300]]{master=true}], reason > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > 208.94.2.146:9300]][discovery/zen/join]; > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > 208.94.2.146:9300]]{master=true}]] > [2012-02-24 10:55:24,561][INFO][discovery.zen] [img700] > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > [inet[/208.94.2.146:9300]]{master=true}], reason > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > 208.94.2.146:9300]][discovery/zen/join]; > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > 208.94.2.146:9300]]{master=true}]] > [2012-02-24 10:55:27,568][INFO][discovery.zen] [img700] > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > [inet[/208.94.2.146:9300]]{master=true}], reason > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > 208.94.2.146:9300]][discovery/zen/join]; > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > 208.94.2.146:9300]]{master=true}]] > [2012-02-24 10:55:30,778][INFO][discovery.zen] [img700] > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > [inet[/208.94.2.146:9300]]{master=true}], reason > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > 208.94.2.146:9300]][discovery/zen/join]; > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > 208.94.2.146:9300]]{master=true}]] > > This has been going on for 10 minutes or so... Any ideas what might > be wrong? > > -Jack ---
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [February 26, 2012, 4:01am UTC](https://discuss.elastic.co/t/master-election-issue/6802/3 "2012-02-26T04:01:07Z")
Speaking of which, I am using master as datanode also, and when first master started it immediately started recovery, causing the discovery of other datanodes to be very slow. I am also using local gateway, so I was wondering if there were any ability to configure start up of initial number (quorum) of datanodes in the cluster before recovery starts? -Jack On Feb 25, 7:56 pm, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > I was able to resolve this by shutting down all nodes in the cluster, > and starting up masters first. > > -Jack > > On Feb 24, 10:57 am,JackLevin[magn...@gmail.com](mailto:magn...@gmail.com) wrote: > > > After full cluster restart, my master is unable to elect itself, here > > are some logs: > > > [2012-02-24 10:55:21,549][INFO][discovery.zen] [img700] > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > [inet[/208.94.2.146:9300]]{master=true}], reason > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > 208.94.2.146:9300]][discovery/zen/join]; > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > 208.94.2.146:9300]]{master=true}]] > > [2012-02-24 10:55:24,561][INFO][discovery.zen] [img700] > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > [inet[/208.94.2.146:9300]]{master=true}], reason > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > 208.94.2.146:9300]][discovery/zen/join]; > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > 208.94.2.146:9300]]{master=true}]] > > [2012-02-24 10:55:27,568][INFO][discovery.zen] [img700] > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > [inet[/208.94.2.146:9300]]{master=true}], reason > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > 208.94.2.146:9300]][discovery/zen/join]; > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > 208.94.2.146:9300]]{master=true}]] > > [2012-02-24 10:55:30,778][INFO][discovery.zen] [img700] > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > [inet[/208.94.2.146:9300]]{master=true}], reason > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > 208.94.2.146:9300]][discovery/zen/join]; > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > 208.94.2.146:9300]]{master=true}]] > > > This has been going on for 10 minutes or so... Any ideas what might > > be wrong? > > > -Jack ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 26, 2012, 7:47pm UTC](https://discuss.elastic.co/t/master-election-issue/6802/4 "2012-02-26T19:47:52Z")
I am not really sure what happened in our process. Regarding the initial mail, you did a full cluster restart, and then got into this state? Do you have special settings per node? Regarding your last question, you can configure the gateway.recover\_after\_nodes setting which will cause the initial recovery process to happen once those nodes are met.[https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml#L253](https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml#L253). On Sunday, February 26, 2012 at 6:01 AM, Jack Levin wrote: > Speaking of which, I am using master as datanode also, and when first > master started it immediately started recovery, causing the discovery > of other datanodes to be very slow. I am also using local gateway, so > I was wondering if there were any ability to configure start up of > initial number (quorum) of datanodes in the cluster before recovery > starts? > > -Jack > > On Feb 25, 7:56 pm, Jack Levin \<[magn...@gmail.com](mailto:magn...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > I was able to resolve this by shutting down all nodes in the cluster, > > and starting up masters first. > > > > -Jack > > > > On Feb 24, 10:57 am,JackLevin\<[magn...@gmail.com](mailto:magn...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > > > After full cluster restart, my master is unable to elect itself, here > > > are some logs: > > > > > [2012-02-24 10:55:21,549][INFO][discovery.zen] [img700] > > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > > [inet[/208.94.2.146:9300]]{master=true}], reason > > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > > 208.94.2.146:9300]][discovery/zen/join]; > > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > > 208.94.2.146:9300]]{master=true}]] > > > [2012-02-24 10:55:24,561][INFO][discovery.zen] [img700] > > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > > [inet[/208.94.2.146:9300]]{master=true}], reason > > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > > 208.94.2.146:9300]][discovery/zen/join]; > > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > > 208.94.2.146:9300]]{master=true}]] > > > [2012-02-24 10:55:27,568][INFO][discovery.zen] [img700] > > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > > [inet[/208.94.2.146:9300]]{master=true}], reason > > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > > 208.94.2.146:9300]][discovery/zen/join]; > > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > > 208.94.2.146:9300]]{master=true}]] > > > [2012-02-24 10:55:30,778][INFO][discovery.zen] [img700] > > > failed to send join request to master [[img700][UO6rSvnqSl-eyoygRj-99g] > > > [inet[/208.94.2.146:9300]]{master=true}], reason > > > [org.elasticsearch.transport.RemoteTransportException: [img700][inet[/ > > > 208.94.2.146:9300]][discovery/zen/join]; > > > org.elasticsearch.ElasticSearchIllegalStateException: Node [[img700] > > > [rk3NK\_UyTpusT4FnXt3\_MQ][inet[/208.94.2.146:9300]]{master=true}] not > > > master for join request from [[img700][rk3NK\_UyTpusT4FnXt3\_MQ][inet[/ > > > 208.94.2.146:9300]]{master=true}]] > > > > > This has been going on for 10 minutes or so... Any ideas what might > > > be wrong? > > > > > -Jack ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:38am UTC](https://discuss.elastic.co/t/master-election-issue/6802/5 "2017-07-06T03:38:11Z")