Can't see the second node whilst attempting to build a cluster

[sc1 (Stephen)]

Hello,

I am looking to add a second node and create a cluster. I have installed version 6.5.1 of Elasticsearch on 2 servers. The primary node has the full stack (Elasticsearch, Kibana, Logstash) and the second node has Elasticsearch installed.

Primary node IP: 192.168.0.64 & Secondary node IP: 192.168.0.139

I have looked at the following questions: Example and haven't been able to find a solution.

There is currently no errors from the logs and everything should be working fine but via Kibana I can't see the secondary node, at all. Networking wise, the two hosts are able to communicate and both configs are near enough identical. The cluster name is identical as well.

In the Discovery section of the YML file:

Primary Node

discovery.zen.ping.unicast.hosts: ["192.168.0.64","192.168.0.139"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.

Secondary Node

discovery.zen.ping.unicast.hosts: ["192.168.0.64","192.168.0.139"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#

On the Primary Node, when I have uncommented discovery.zen.minimum_master_nodes it actually doesn't start, I have been looking at the documentation but can't find.

Could someone help me understand why the services are running fine, but I can't seem to get the cluster working as only 1 node is appearing in Kibana.

[DavidTurner]

You must set discovery.zen.minimum_master_nodes: 2 on both nodes in order for this to work correctly. However, you say that you tried this and "it actually doesn't start". What do you mean? What did the log messages say?

[sc1 (Stephen)]

Thanks for the swift response @DavidTurner

I have set the discovery.zen.minimum_master_nodes to 2 as suggested. When I do this however the following starts:

root@main_node:~# sudo service kibana status
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-01-07 12:56:32 GMT; 1min 21s ago
 Main PID: 2886 (node)
    Tasks: 10
   Memory: 279.4M
      CPU: 6.440s
   CGroup: /system.slice/kibana.service
           └─2886 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jan 07 12:57:46 main_node kibana[2886]: exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."}
Jan 07 12:57:46 main_node kibana[2886]: {"type":"response","@timestamp":"2019-01-07T12:57:46Z","tags":[],"pid":2886,"method":"get","statusCode":500,"req":{"url":"/favicon.ico","method":"get","headers":{"connection":"upgrade","host":"main_node","pragm
Jan 07 12:57:47 main_node kibana[2886]: {"type":"log","@timestamp":"2019-01-07T12:57:47Z","tags":["info","authentication"],"pid":2886,"message":"Authentication attempt failed: [exception] Security must be explicitly enabled when using a trial license. Enable security
Jan 07 12:57:47 main_node kibana[2886]: {"type":"error","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"level":"error","error":{"message":"[exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security
Jan 07 12:57:47 main_node kibana[2886]: eption] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."}
Jan 07 12:57:47 main_node kibana[2886]: {"type":"response","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"method":"get","statusCode":500,"req":{"url":"/app/kibana","method":"get","headers":{"connection":"upgrade","host":"main_node","cache-
Jan 07 12:57:47 main_node kibana[2886]: {"type":"log","@timestamp":"2019-01-07T12:57:47Z","tags":["info","authentication"],"pid":2886,"message":"Authentication attempt failed: [exception] Security must be explicitly enabled when using a trial license. Enable security
Jan 07 12:57:47 main_node kibana[2886]: {"type":"error","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"level":"error","error":{"message":"[exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security
Jan 07 12:57:47 main_node kibana[2886]: exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."} 

So I went in the main node YML file and added the following:

xpack.security.enabled: true

As suggested in the official documentation: Enable Security but as far as I was aware, this would be a paid feature with Xpack which I am not yet looking to get into as I am still testing and learning.

When I add the line, this is the error I see:

root@main_node:~# sudo service kibana status
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-01-07 13:00:19 GMT; 2min 0s ago
 Main PID: 3205 (node)
    Tasks: 10
   Memory: 272.0M
      CPU: 5.740s
   CGroup: /system.slice/kibana.service
           └─3205 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST r
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:graph@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST re
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST r
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:security@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for RES
Jan 07 13:01:25 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:25Z","tags":["license","warning","xpack"],"pid":3205,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_
Jan 07 13:01:55 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:55Z","tags":["license","warning","xpack"],"pid":3205,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_

Elasticsearch appears to be running with no issues in the logs on both nodes but I could be wrong.

[DavidTurner]

Ok, if the Elasticsearch logs seem to be clear then I'm not the best person to help. Can you try GET _cluster/health from the command line (i.e. something like curl http://192.168.0.64:9200/_cluster/health) to confirm that Elasticsearch really is running and has successfully formed a cluster of 2 nodes? If it has, it's probably best to ask about these Kibana messages in the Kibana forum.

[sc1 (Stephen)]

Hey @DavidTurner

The only changes I have done are the Discovery Zen section, is there somewhere else I am supposed to change ? For example, the documentation mentions changing nodes to Master/Data nodes/Ingest nodes but I can't see any of that in my default yml file?

I changed the discovery_zen.minimum_master_nodes back to 1 and my main node works again and doesn't throw any of the above errors, it's only when I change it to 2 it shows errors. So I think the issue is still in just Elasticsearch.

[DavidTurner]

The errors above are from Kibana's log, which is why I think you should ask about them in the Kibana forum. If you think Elasticsearch is not working right now, it'd be best to take Kibana out of the equation for now and just focus on the Elasticsearch part.

@sc1

I changed the discovery_zen.minimum_master_nodes back to 1

It absolutely must be 2 if you have 2 master-eligible nodes in your cluster.

[sc1 (Stephen)]

I have asked the following question on the Kibana forum. As I am still facing the issue, any help would be greatly appreciated.

Question in Kibana Forum

[tsullivan (Tim Sullivan)]

Hi, the messages you were seeing that you had to explicitly enable Security might have been showing up because you had xpack.security settings in your elasticsearch.yml file. ES warned you that it can't do anything with those settings because Security wasn't automatically enabled for your Trial license.

If that was the case, you don't need to enable security, just take out the settings from your elasticsearch.yml that try to configure security features.

From there on, it looks like Kibana was failing to start up completely because it could not authenticate with Elasticsearch. That could have happened because no users were created in ES, or kibana.yml did not have the auth config lines (elasticsearch.username, elasticsearch.password) or it had the incorrect auth. Since it says missing authentication token I think that means you had no auth settings in your kibana.yml

Other than node.master: true in one node and node.data: true in another, is everything in the ES configs exactly the same? Since security is popping up as a trouble spot, you might have had different security settings in one of the configs.

Networking wise, the two hosts are able to communicate and both configs are near enough identical

How do you determine the network status? To verify what you say, you'd need to check:

curl https://elasticsearch:9200/_cluster/health

and look for:

  "number_of_nodes": 2

in the response. Invalid security settings will certainly prevent a node from joining a cluster, so check for that.

Hope that helps!

[sc1 (Stephen)]

Good afternoon Tim, thank you for the response.

I have gone through what you've said and can confirm that xpack.security settings are all off on all 3 elasticsearch.yml files. As I am using the same version 6.5.4 on all 3 servers, the .yml is identical on all 3.

In regards to verifying the network, I had been able to telnet to all hosts and that showed that the network was working. I have however used the method you suggested and here are the results:

The following results are from the two nodes I am trying to add, where I seem to have no issues with Elasticsearch running.

{"cluster_name":"adfs-logs","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

And from the second node:

{"cluster_name":"adfs-logs","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@lcs-admqdjg-pc elasticsearch]

When I run the command on the main server:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Kibana is still throwing the license errors, but literally, when what's frustrating is it seems to want to function just as a single node. Let me know if there's more logs you require.

[tsullivan (Tim Sullivan)]

Hi, this info is great! I formatted the JSON with jq and started looking.

When you have 2 nodes joined in a cluster, they both report:

"number_of_nodes": 2,
"number_of_data_nodes": 2,

Something is wrong here. There always needs to be an elected master in the cluster so each data node can be notified of updates to the cluster state. You have both nodes as data nodes, instead of either being elected as master.

Both nodes also show:

"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,

The problem that's stopping Kibana from functioning doesn't look like a License issue. Without an elected master node, you will not be able to create an index and without any shards, there is nothing to query. I think the 3rd part you captured is to that effect. This: master_not_discovered_exception needs to be resolved.

I'd like you to take a look more at the documentation on the ES node types, especially master node: Node | Elasticsearch Guide [8.11] | Elastic

If that's not helping, let's get as much of the elasticsearch.yml contents as you can post here (no confidential info please).

We'd have to look for any settings that are preventing the nodes from being master-eligible.

You have nothing like this in your elasticsearch.yml, right?

  node.master:false

Or could there be any possible firewall issues? The nodes need to be able to talk to each other in TCP port 9300.

[tsullivan (Tim Sullivan)]

BTW what does /_cluster/health show when you have just a single node?

When you say it is working with the 1 node, does that mean you are able to index data and search on it (either in curl or in Kibana)?

[sc1 (Stephen)]

Hi Tim, apologies for the delayed response.

When running as a single node, I face 0 issues. I am able to access via Kibana, process logs, build dashboards etc.

Here is the elasticsearch.yml as requested:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: adfs-logs
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: "test-node-1"
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: localhost
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["XXX.XXX.XXX.64", "XXX.XXX.XXX..63:9300", "XXX.XXX.XXX.139:9300"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

xpack.security.enabled: false

And the curl request you asked for when it's working:

"cluster_name":"adfs-logs","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":716,"active_shards":716,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":703,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.45806906272022}

Let me know if there's anything else you need.

[tsullivan (Tim Sullivan)]

In your elasticsearch.yml, you have:

network.host: localhost

That is making Elasticsearch listen just on the loopback interface: it'll only be able to connect to requests that were made to localhost. The nodes will need to bind to a non-loopback address. See https://www.elastic.co/guide/en/elasticsearch/reference/current/network.host.html

[sc1 (Stephen)]

Good afternoon Tim,

I followed said advice and removed network.host: localhost on all the yml files and this has resolved the problem. The cluster is now live!!!! Can't thank you enough.

[tsullivan (Tim Sullivan)]

Fantastic! Sorry it took so long to get there, but glad it worked out!