Missing Client Nodes in v5.6

Elastic Stack Elasticsearch
21

@iam.Carrot Are you talking about that elasticsearch is in inactive state ?Sorry, I am not clear about your point. Can you please elaborate this?

And as you told ports are open but your nodes are still not communicating in the cluster for this firstly check some points :
a) Elasticsearch service is in running state on your all nodes.
b) if it size problem to share yml files for all please check at your end : cluster name and host name in all nodes file should same and also make sure one of node have node.,master:true parameter.

share elasticsearch and logstash logs for more investigating the issue.

22

it seems that there is some issue with your configuration. Some thing you have changed in only one node .
Please share your node configuration file if there is size problem can you share a snapshot of this?

23

@poojagupta thanks for staying on for helping me out, I really appreciate the help. I've uploaded the updated yml files to onedrive here. The files are prepended with the names as master and client based on the names of the nodes es-master-01 and es-client-01 I've used the private ip addresses of the nodes to connect. When I use the public ip address, elastic fails to start.

Elasticsearch is running on both the nodes, I can telnet to port 9300 and 9200 for each of em across nodes.

The issue with logstash was that I had to put it the inactivity timeout in the config file for logstash to work. I was using filebeat to push logs from a file, to logstash pipeline to push the logs into elastic and view onto kibana. I'll share the config files as soon as you tell me which all files do you want to see.

24

@iam.Carrot After looking at your configuration file of client and master node i observe some parameter needs to be change.

  1. firstly all node ip should be included in discovery.unicast parameter like for master node include masternode ip also. same as client node.
  2. for client node mention the node.client:true parameter also.

just to confirm that there is no data node in your cluster for performing the operation you have to keep one data node atleast for this you can make master node as data node as well . change the node.data: false to true.

Also why you are using IP: 22.0.66.X instead of localhost or hostname .. you can mention network.host: 0.0.0.0 for all nodes.
it would be better for a cluster and after this your all nodes will joiin with that ip.
if your nodes are able to resolve host then it should be pingable via hostname from each other nodes. for this make sure you have entries for all nodes in etc/hosts file .
after this you don't need to mention the ip anywhere and for unicast parameter you can mention the hostname for all nodes to discover each other node.

In last , i am requesting to share the logs for elasticsearch and logstash (as you are using it for logs) for looking the issue in depth (if still u faced the issue)

25

Why are you setting discovery.zen.minimum_master_nodes: 0? That is not a valid setting.

26

@poojagupta The issue with this setting is that, in the version 5.6, they've changed the client to coordinate only node and the node.client leads to elasticsearch not running as node.client is not a valid setting. If you have a look at the chat earlier, @Christian_Dahlqvist had told that set the discovery.zen.minimum_master_nodes: 0 as it's a small cluster and I shouldn't be going for a dedicated master so I did.

I've made the changes as you say, unfortunately there's no luck. The things remain the same. I've uploaded the yml files here. I've also uploaded the logs that were generated after the changes just so that you have a clearer picture.

27

All clusters need at least one, but ideally three, master-eligible node(s). Without this a cluster can not function. By default all nodes have all roles (hold data and are master-eligible) and the minimum_master_nodes parameter need to be set to the correct value (majority of master-eligible nodes in cluster). For small clusters it generally makes sense to have all nodes be equal and have all roles. Using dedicated node types is usually not required until your cluster grows.

Start up all the nodes and verify that they are running and that you can connect to them locally. Make sure that you have at least one data node. As mentioned above I would recommend you to start up all nodes with default roles. You should be able to telnet to port 9300 from the local host without problems. Then try to telnet to the 9300 port from the other nodes. Make sure that all nodes can connect to each other.

28

@Christian_Dahlqvist could you please recommend the number of nodes untill I should switch to dedicated roles for the nodes. for example: untill I have less than 4 nodes, all nodes should have all rodes.

29

As long as you only have 3-5 data nodes I would not necessarily bother about dedicated node types, unless you are having issues due to heavy load.

30

@iam.Carrot : Sorry i didn't notice your previous chat for which version you are working. But one more point as you told its small cluster and there is no dedicated master node then comment out this line in the file.

31

Commenting out the line will make it default to 1, which is only the correct setting if you have only one master-eligible node in the cluster.

32

@Christian_Dahlqvist great. Thank you for that information. I should switch to all nodes acting all roles by simply not setting any node.* in the yml file. Also should set the minimum_master_nodes to 0?

33

No. Please follow these instructions.

34

@Christian_Dahlqvist Great thank for all the help. I just hope my cluster forms too. It's been quite a while i've been spending on this and simply just won't work

35

I agree with your point. But i am little bit surprise if we don't mention that line in the file or will mention that value to 1 not 0, then there is not cause of the issue which @iam.Carrot is facing.
i don't still get taht is it real cause of this issue

36

Setting it to 0 is apparently wrong, so I wanted to correct that. I suspect the issue is related to network, which is why I suggested testing connectivity between all nodes using telnet.

37

@Christian_Dahlqvist @poojagupta Believe me, I've tried. I can't find any networking issues. I can telnet to port 9200 and 9300 from one node to another, I can curl from one node to another. I took both the nodes and set their minimum_master_nodes to 2 in a cluster of 2 using the formula (number of master eligible /2 +1) => 2/2 +1 i.e 2. Now neither of them would response as both have no master. I set one node as node.master: true. Still they both won't respond when minimum_master_nodes is set to 2.

Should I provide a TeamViewer access? would that help?

38

please do one thing. create a new cluster name and start again elasticsearch service on both nodes.
There may be reason for data issue.

39

Change both nodes so that they have default configuration (are master-eligible and can hold data), with minimum_master_nodes set to 2 and all nodes listed in the discovery.zen.ping.unicast.hosts list. See if this makes a difference. See if this makes any difference and look in the Elasticsearch logs if it doesn't help.

40

@poojagupta @Christian_Dahlqvist I am so happy and glad to inform that the cluster has now been formed and all things are working perfectly fine.

What was the issue?

The issue was i had replicated the nodes after I had installed elasticsearch because of which I had all nodes with the same id and all we're acting as a client. I deleted the contents of the directory /var/lib/elasticsearch and did a elasticsearch restart. now it works perfectly fine. Below is the success code line:

[root@es-client-01 ec2-user]# curl 22.0.6.21:9200/_cat/nodes?v
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
22.0.6.64            6          33   0    0.00    0.03     0.01 mdi       *      es-master-01
22.0.6.21            6          33   0    0.00    0.01     0.00 mdi       -      es-client-01
[root@es-client-01 ec2-user]#

I have no idea how I found it, I was just going for it to cover all bases.

You guys have been of great help. Thanks a lot, I really appreciate it.

1 Like