Data node can't find master node ,but master can find data node

Elastic Stack Elasticsearch
1

here is log info as below:

[2016-07-29 00:05:43,764][INFO ][cluster.service ] [data_node_144] detected_master {master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}, added {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true},}, reason: zen-disco-receive(from master [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}])
[2016-07-29 00:05:43,915][INFO ][discovery.zen ] [data_node_144] master_left [{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true}], reason [transport disconnected]
[2016-07-29 00:05:43,974][WARN ][discovery.zen ] [data_node_144] master left (reason = transport disconnected), current nodes: {{data_node_144}{LkMlU7XeSqC-j0K_o05iUw}{192.168.201.144}{192.168.201.144:9300}{master=false},{data_node_148}{FKdVDjWjQ4eykqkMNRNxUQ}{192.168.201.148}{192.168.201.148:9300}{master=false},{master_node_102}{zQR04axjRxGFhmVxw3N4Pg}{192.168.201.102}{192.168.201.102:9300}{data=false, master=true},{master_node_103}{v83JEFKaQa6gXrG8o5xfBw}{192.168.201.103}{192.168.201.103:9300}{data=false, master=true},{data_node_145}{dF-4DvGlT22v2vI68PrIkQ}{192.168.201.145}{192.168.201.145:9300}{master=false},{data_node_146}{TDPHZWaRRm-lTfM2EM3bPQ}{192.168.201.146}{192.168.201.146:9300}{master=false},{master_node_101}{eqxEaFh8TeqR4VmhKPdT_g}{192.168.201.101}{192.168.201.101:9300}{data=false, master=true},{data_node_147}{-evmqt_nSV-RSeXTZ2w15w}{192.168.201.147}{192.168.201.147:9300}{master=false},}
[2016-07-29 00:05:43,974][INFO ][cluster.service ] [data_node_144] removed {{master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true},}, reason: zen-disco-master_failed ({master_node_200}{KPQpU4cdRyiqCT488ZFglg}{192.168.201.200}{192.168.201.200:9300}{data=false, master=true})
[2016-07-29 00:05:43,974][WARN ][cluster.service ] [data_node_144] failed to notify ClusterStateListener
java.lang.IllegalStateException: master not available when registering auto-generated license
at org.elasticsearch.license.plugin.core.LicensesService.requestTrialLicense(LicensesService.java:749)
at org.elasticsearch.license.plugin.core.LicensesService.clusterChanged(LicensesService.java:483)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

and the configure of data node:

cluster.name: es_cluster
node.name: data_node_144
node.master: false
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
network.host: 0.0.0.0
network.publish_host: 0.0.0.0
gateway.recover_after_nodes: 2
discovery.zen.ping_timeout: 60s
discovery.zen.ping.unicast.hosts: ["192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.1
03","192.168.201.144","192.168.201.145","192.168.201.146","192.168.201.147","192.168.201.148" ]
discovery.zen.minimum_master_nodes: 3

configure of master node:

cluster.name: es_cluster
node.name: master_node_200
node.master: true
node.data: false
path.data: /data/es/data
path.logs: /data/es/logs
network.host: 0.0.0.0
network.publish_host: 0.0.0.0
gateway.recover_after_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping_timeout: 60s
discovery.zen.ping.unicast.hosts: ["192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.103" ]
discovery.zen.minimum_master_nodes: 3
bootstrap.mlockall: true

it's weird that master node can find the data node.
PS. master node : "192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.103",192.168.201.200" and others are data node.

2

There is no need to put all data nodes into this list, all nodes in your cluster need only

What is the size of your cluster, and why you need 4 master nodes? Shouldn't 3 of them be enough?

3
  1. 5 data nodes and 4 master nodes,total 130G+ data size in my cluster.
  2. i think 4 master nodes can reduce the possibility of Split Brain.
  3. i don't have an idea of how many master nodes should i set,maybe should i set one master node to a client node?
    :joy:
4

You only really need 3 masters to form a quorum.

5

hi,i have change one master node to client node and restart 5 data nodes,the issue is still exist.:sob:

6

3 master nodes are enough, and set discovery.zen.minimum_master_nodes: to 2.

How many data nodes can't join the cluster? If it's only one node, I would rather forget about that node and set up another one to replace the failed one.

Is the failed node a new one that you are trying to put into the cluster? Could you try to install/reinstall the license plugin on all nodes? Has the trial license expired?

7
  1. i have set discovery.zen.minimum_master_nodes: to 2,it still cannot work.
  2. all 5 data nodes ocurred the issue.
  3. i only explicitly install license plugin in 5 data nodes,don't install in master nodes,this could be the poiont?
8

License plugin must be installed on all nodes: client, master, data.

1 Like
9

thank you very much,one client node work now after i install license plugin on it and restart.
i will try other nodes.:kissing_closed_eyes:

1 Like
10

The trial license will expire after a month, you need to request a free license and update for your cluster.

11

btw,

discovery.zen.ping.unicast.hosts: ["192.168.201.200","192.168.201.101","192.168.201.102","192.168.201.103","192.168.201.144","192.168.201.145","192.168.201.146","192.168.201.147","192.168.201.148" ]
what exactly should i set?
Should i only set master nodes ip in this field? Or other data nodes ip in this field.
And i have another question,should i set the filed "discovery.zen.minimum_master_nodes: 3" in non-master nodes?of course and the field "gateway.recover_after_nodes: 2" and someelse that iS master‘s functionalities.

12

i don't know how to get a free license, i install it througth the command "plugin install license" all the time before.

13

You can request a free basic license here https://register.elastic.co/marvel_register

14

i want to know if all nodes uninstall the license plugin,and there is no need to install it,is it?

15

If you don't need marvel, I don't think that you need license plugin.

16

thank you for your answers,have a nice day,it's afternoon in my location.:kissing_heart: