Data node can't find master node ,but master can find data node

here is log info as below:

[2016-07-29 00:05:43,764][INFO ][cluster.service ] [data_node_144] detected_master {master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true}, added {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true},}, reason: zen-disco-receive(from master [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true}])
[2016-07-29 00:05:43,915][INFO ][discovery.zen ] [data_node_144] master_left [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true}], reason [transport disconnected]
[2016-07-29 00:05:43,974][WARN ][discovery.zen ] [data_node_144] master left (reason = transport disconnected), current nodes: {{data_node_144}{LkMlU7XeSqC-j0K_o05iUw}{}{}{master=false},{data_node_148}{FKdVDjWjQ4eykqkMNRNxUQ}{}{}{master=false},{master_node_102}{zQR04axjRxGFhmVxw3N4Pg}{}{}{data=false, master=true},{master_node_103}{v83JEFKaQa6gXrG8o5xfBw}{}{}{data=false, master=true},{data_node_145}{dF-4DvGlT22v2vI68PrIkQ}{}{}{master=false},{data_node_146}{TDPHZWaRRm-lTfM2EM3bPQ}{}{}{master=false},{master_node_101}{eqxEaFh8TeqR4VmhKPdT_g}{}{}{data=false, master=true},{data_node_147}{-evmqt_nSV-RSeXTZ2w15w}{}{}{master=false},}
[2016-07-29 00:05:43,974][INFO ][cluster.service ] [data_node_144] removed {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true},}, reason: zen-disco-master_failed ({master_node_200}{KPQpU4cdRyiqCT488ZFglg}{}{}{data=false, master=true})
[2016-07-29 00:05:43,974][WARN ][cluster.service ] [data_node_144] failed to notify ClusterStateListener
java.lang.IllegalStateException: master not available when registering auto-generated license
at org.elasticsearch.license.plugin.core.LicensesService.requestTrialLicense(
at org.elasticsearch.license.plugin.core.LicensesService.clusterChanged(
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(
at org.elasticsearch.cluster.service.InternalClusterService$
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

and the configure of data node: es_cluster data_node_144
node.master: false true /data/es/data
path.logs: /data/es/logs
gateway.recover_after_nodes: 2
discovery.zen.ping_timeout: 60s ["","","","
03","","","","","" ]
discovery.zen.minimum_master_nodes: 3

configure of master node: es_cluster master_node_200
node.master: true false /data/es/data
path.logs: /data/es/logs
gateway.recover_after_nodes: 2 false
discovery.zen.ping_timeout: 60s ["","","","" ]
discovery.zen.minimum_master_nodes: 3
bootstrap.mlockall: true

it's weird that master node can find the data node.
PS. master node : "","","",""," and others are data node.

There is no need to put all data nodes into this list, all nodes in your cluster need only

What is the size of your cluster, and why you need 4 master nodes? Shouldn't 3 of them be enough?

  1. 5 data nodes and 4 master nodes,total 130G+ data size in my cluster.
  2. i think 4 master nodes can reduce the possibility of Split Brain.
  3. i don't have an idea of how many master nodes should i set,maybe should i set one master node to a client node?

You only really need 3 masters to form a quorum.

hi,i have change one master node to client node and restart 5 data nodes,the issue is still exist.:sob:

3 master nodes are enough, and set discovery.zen.minimum_master_nodes: to 2.

How many data nodes can't join the cluster? If it's only one node, I would rather forget about that node and set up another one to replace the failed one.

Is the failed node a new one that you are trying to put into the cluster? Could you try to install/reinstall the license plugin on all nodes? Has the trial license expired?

  1. i have set discovery.zen.minimum_master_nodes: to 2,it still cannot work.
  2. all 5 data nodes ocurred the issue.
  3. i only explicitly install license plugin in 5 data nodes,don't install in master nodes,this could be the poiont?

License plugin must be installed on all nodes: client, master, data.

1 Like

thank you very much,one client node work now after i install license plugin on it and restart.
i will try other nodes.:kissing_closed_eyes:

1 Like

The trial license will expire after a month, you need to request a free license and update for your cluster.

btw, ["","","","","","","","","" ]
what exactly should i set?
Should i only set master nodes ip in this field? Or other data nodes ip in this field.
And i have another question,should i set the filed "discovery.zen.minimum_master_nodes: 3" in non-master nodes?of course and the field "gateway.recover_after_nodes: 2" and someelse that iS master‘s functionalities.

i don't know how to get a free license, i install it througth the command "plugin install license" all the time before.

You can request a free basic license here

i want to know if all nodes uninstall the license plugin,and there is no need to install it,is it?

If you don't need marvel, I don't think that you need license plugin.

thank you for your answers,have a nice day,it's afternoon in my location.:kissing_heart: