How can a configure two node in one cluster

dattack13 · March 6, 2019, 11:00am

Hi,

I have situation where I have installed two instance of Elasticsearch setup in remote server 1 (node1) and remote server 2 (node2), and I want to connect both these node in one cluster, however I also installed the kibana in one server and when I run the GET _cluster/health I get

{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Can you please help while providing any sample for both config for node1 and node2, and how can I confirm both nodes are added in one cluster.

Thanks
Gaurav Singh

dattack13 · March 6, 2019, 11:12am

Below is my elasticsearch.yml file in server 1

bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 2
node.name: Server1
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["Server1","Server2"]
network.host: Server1

Server2 : elasticsearch.yml file -

bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 1
node.name: Server2
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false
discovery.zen.ping.unicast.hosts: ["Server1","Server2"]
network.host: Server2

Thanks
Gaurav Singh

dadoonet · March 6, 2019, 11:27am

Here is what I would use (simplified version, corrected version, formatted):

Node 1:

cluster.name: elasticsearch
node.name: server1
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
discovery.zen.ping.unicast.hosts: ["ip_of_server1","ip_of_server2"]
network.host: ip_of_server1

Node 2:

cluster.name: elasticsearch
node.name: server2
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
discovery.zen.ping.unicast.hosts: ["ip_of_server1","ip_of_server2"]
network.host: ip_of_server2

dattack13 · March 6, 2019, 11:41am

Hi,

While I am trying to run the _cluster/health I am getting error -

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Thanks
Gaurav Singh

dadoonet · March 6, 2019, 11:57am

Could you share both full logs of both machines?

dattack13 · March 6, 2019, 12:27pm

FYI

[2019-03-06T06:23:06,975][WARN ][o.e.d.z.ZenDiscovery ] [APVEDxxxxx] not enough master nodes discovered during pinging (found [[Candidate{node={APVEDxxxxx}{5R-udMJIQPe2y9KByEoC2Q}{uDRsGmGcTJyF0SQME9vdMQ}{apvedxxxxx}{10.xx.xxx.xxx:9300}{ml.machine_memory=17179398144, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

Thanks
Gaurav Singh

dattack13 · March 6, 2019, 12:35pm

Server 2 Log is below -

org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.1.jar:6.6.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]
[2019-03-06T06:29:40,292][INFO ][o.e.d.z.ZenDiscovery     ] [APVEDSERVER2] failed to send join request to master [{APVEDSERVER1}{5R-udMJIQPe2y9KByEoC2Q}{uDRsGmGcTJyF0SQME9vdMQ}{APVEDSERVER1}{10.xx.yyy.zzz:9300}{ml.machine_memory=17179398144, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [ElasticsearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.]; ]

Thanks
Gaurav Singh

dadoonet · March 6, 2019, 12:43pm

Could you share both full logs of both machines?

dattack13 · March 6, 2019, 5:56pm

After modifying the line :

discovery.zen.minimum_master_nodes: 1

It worked

Thanks
Gaurav Singh

dadoonet · March 6, 2019, 5:58pm

I'd now recommend starting a 3rd node (master node only) and revert your modification to:

discovery.zen.minimum_master_nodes: 2

On all nodes.

dattack13 · March 6, 2019, 6:01pm

So you are saying I should have atleast 3 nodes and I have to make minimum two as master node, Am I correct?

dadoonet · March 6, 2019, 6:03pm

That's the recommandation to avoid split brain issues.
All 3 nodes must be master elligible nodes. Basically

Node 1:

cluster.name: elasticsearch
node.name: server1
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
discovery.zen.ping.unicast.hosts: ["ip_of_server1","ip_of_server2","ip_of_server3"]
network.host: ip_of_server1
discovery.zen.minimum_master_nodes: 2

Node 2:

cluster.name: elasticsearch
node.name: server2
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
discovery.zen.ping.unicast.hosts: ["ip_of_server1","ip_of_server2","ip_of_server3"]
network.host: ip_of_server2
discovery.zen.minimum_master_nodes: 2

Node 3:

cluster.name: elasticsearch
node.name: server3
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
discovery.zen.ping.unicast.hosts: ["ip_of_server1","ip_of_server2","ip_of_server3"]
network.host: ip_of_server3
discovery.zen.minimum_master_nodes: 2
# Optionally depending if you want also to store data on it or not
node.data: false

dattack13 · March 6, 2019, 6:23pm

Thank You so much

system · April 3, 2019, 6:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.