Concept of Elasticsearch cluster

Dominic_Page · October 7, 2019, 3:00pm

Hi @pyerunka

Do you get the same response if you send:

curl -X GET abcd.com:9300

when logged into the Master node?

pyerunka · October 9, 2019, 5:01am

Hello @Dominic_Page,

No. I got below response:
curl: (7) Failed to connect to abcd.com port 9300: Timed out.

Regards,
Priyanka

Dominic_Page · October 9, 2019, 7:00am

Hi @pyerunka,

Can you check again that the Elasticsearch process is still running on the Master Node?

pyerunka · October 9, 2019, 11:31am

Hello @Dominic_Page,

Yes, Elasticsearch process is still running on the Master Node on port 9200.
It shows me below record:

{
"name" : "Master-node",
"cluster_name" : "Cluster",
"cluster_uuid" : "gzXXocd1T7WVqeNeZrwLKw",
"version" : {
"number" : "7.2.0",
"build_flavor" : "default",
"build_type" : "zip",
"build_hash" : "508c38a",
"build_date" : "2019-06-20T15:54:18.811730Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Regards,
Priyanka Yerunkar.

Dominic_Page · October 9, 2019, 2:20pm

Great!

What was the exact request you made to retrieve this JSON?

pyerunka · October 10, 2019, 3:28am

Hello @Dominic_Page,

I used following command: "curl -X GET abcd.com:9200 "

Regards,
Priyanka

Tek_Chand · October 10, 2019, 5:31am

@pyerunka,

Seems that you are using some firewall on your machine. So you need open port 9300 on both master and data node.

Your node will communicate over port 9300.

Thanks.

pyerunka · October 10, 2019, 6:14am

Hello @Tek_Chand,

Thanks for reply!!
I have raised an request to our network team to open a firewall between two servers on port 9300.

Regards,
Priyanka

Dominic_Page · October 10, 2019, 9:32am

Hi @pyerunka

You might also want to check that the hosts are resolving to abcd.com and xyz.com

You could also consider temporarily using the IPs for those if that might be the issue

pyerunka · October 17, 2019, 5:55am

Hello @Dominic_Page,

Earlier i have raised a request to open a firewall between two servers using ports 9300 and 9400.
Still i am not able to start Es on data node.

I am getting below error:

[2019-10-17T05:47:04,195][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-node1] master not discovered yet: have discovered ;
discovery will continue using [136.252.129.115:9200] from hosts providers and [{Data-node1}{vMrL87DfRfmZV46pYXb2HA}{K_MCTNi9RmaA4ugiKPeprQ}{xyz.com}{136.252.130.86:9200}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from
last-known cluster state; node term 0, last-accepted version 0 in term 0

Can you please guide me on this?

Regards,
Priyanka

Christian_Dahlqvist · October 17, 2019, 7:44am

Node discovery should use port 9300, not 9200 which based on the error message is configured.

pyerunka · October 17, 2019, 8:55am

Hello @Christian_Dahlqvist,

Thanks for reply!!!
Can you please explain this more? how do i check so that it will use port 9300?

Regards,
Priyanka

Christian_Dahlqvist · October 17, 2019, 8:59am

Please show your elasticsearch.yml file.

pyerunka · October 17, 2019, 9:01am

Hello @Christian_Dahlqvist,

Please find below Elasticsearch.yml file for master node:

cluster.name: Cluster
node.name: Master-node
node.master: true
node.data: true
network.host: abcd.com
transport.tcp.port: 9300-9400
node.ingest: true
discovery.zen.ping.unicast.hosts: [" abcd.com","xyz.com"]
bootstrap.memory_lock: true
http.port: 9200
cluster.initial_master_nodes: ["abcd"]
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/

and also find Elasticsearch.yml file for data node:

cluster.name: Cluster
node.name: Data-node1
node.master: false
node.data: true
node.ingest: false
network.host: xyz.com
discovery.zen.ping.unicast.hosts: ["abcd.com","xyz.com"]
transport.tcp.port: 9300-9400
transport.host: xyz.com
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/
bootstrap.memory_lock: true
http.port: 9400

Regards,
Priyanka

Dominic_Page · October 17, 2019, 10:18am

hi @pyerunka

Try changing

discovery.zen.ping.unicast.hosts: ["abcd.com","xyz.com"]

to

discovery.seed_hosts: ["abcd.com","xyz.com"]

For both nodes and restarting

pyerunka · October 18, 2019, 2:48am

Hello @Dominic_Page,

I tried by changing discovery.zen.ping.unicast.hosts to discovery.seed_hosts.
Still i am facing a same issue.

Regards,
Priyanka

Dominic_Page · October 18, 2019, 8:15am

Hi @pyerunka

I think we need to simplify the configs, so we can concentrate on the main issue, which is connecting your nodes into a single cluster.

Please can you:

Stop both nodes
On both nodes, delete the entire "./data" folder, so we start up clean
Edit your Master node config to just leave the following parameters, please delete the other parameters:

cluster.name: Cluster
node.name: Master-node
node.master: true
network.host: _global_
cluster.initial_master_nodes: ["Master-node"]

edit your Data node config to delete all extra configuration parameters and just leave the following:

cluster.name: Cluster
node.name: Data-node
node.master: false
network.host: _global_
discovery.seed_hosts: ["abcd.com"]

Start Master-node on server abcd.com and wait until the console logging messages show a message regarding

"Cluster health status"

Make the following request from both servers, either from command line or from a browser

$ curl -X GET http://abcd.com:9200

You should get some JSON back that looks something like this:

{
  "name" : "node1",
  "cluster_name" : "my_cluster",
  "cluster_uuid" : "5F6fjOcaRXCRTyNFFKvXXQ",
  "version" : {
    "number" : "7.3.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "4749ba6",
    "build_date" : "2019-08-19T20:19:25.651794Z",
    "build_snapshot" : false,
    "lucene_version" : "8.1.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

If you DON'T get JSON here, then stop & post the response here

If you DO get JSON then proceed to start the Data-node on the other server xyz.com

Then please post the console output from the Data-node here

Please don't add extra parameters, we need to keep things simple so we can concentrate on the main issue

pyerunka · October 18, 2019, 11:13am

Hello @Dominic_Page,

Start Master-node on server abcd.com and wait until the console logging messages show a message regarding.

--How do i start master node from data node?

Regards,
Priyanka

Dominic_Page · October 18, 2019, 11:16am

Hi @pyerunka from previous comments:

Yes, I am running on 2 separate hosts. The host name of the server where the master node is running is [abcd.com](http://abcd.com/) Data node is not on the same server. The host name of the server where the data node is running is [xyz.com](http://xyz.com/)

Is the Master node still installed on abcd.com?

pyerunka · October 18, 2019, 11:18am

Hello @Dominic_Page,

No master node is not installed on abcd,com.
My master node is on xyz.com and data node is on abcd.com.
You want me to run master node on xyz.com. Am I right?

Regards,
Priyanka