Elasticsearch Cluster: How to ensure node communication


(Rahul Nama) #1

Hello

Please have a look at ES configuration and let me know what's wrong

Node:1

     node.name: "es-node-1"
     
    node.master: true
     
    node.data: true
     
    network.host: 10.0.0.74
    http.port: 9200
     
    discovery.zen.ping.unicast.hosts: ["10.0.0.30","10.0.0.74"]



Node 2:

    node.name: "es-node-2"
     
    node.master: true
     
    node.data: true
     
    network.host: 10.0.0.30
    http.port: 9200
     
    discovery.zen.ping.unicast.hosts: ["10.0.0.30","10.0.0.74"]

When I hit: (http://10.0.0.30:9200/_cluster/health/?level=shards?pretty)

response:

{"cluster_name":"my-cluster","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}

number of nodes is showing 1 instead of 2.

did I miss anything in config file?

Rahul


(Tim Vernum) #2

Those config file do not contain cluster names, but I can see from your Rest response that your 10.0.0.30 node (Node 2) thinks its cluster name is "my-cluster".

What is the full configuration that you are using for these nodes?


(Rahul Nama) #3

Hey @TimV

this is my full configuration

bootstrap.memory_lock: true
cluster.name: my-cluster
 
node.name: "es-node-1"
 
node.master: true
 
node.data: true
 
network.host: 10.0.0.74
http.port: 9200
 
discovery.zen.ping.unicast.hosts: ["10.0.0.30:9200","10.0.0.74"]
thread_pool.bulk.queue_size: 2000
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs

(Tim Vernum) #4

What about the other node?


(Rahul Nama) #5

@TimV

please have a look

bootstrap.memory_lock: true
cluster.name: my-cluster
 
node.name: "es-node-2"
 
node.master: true
 
node.data: true
 
network.host: 10.0.0.30
http.port: 9200
 
discovery.zen.ping.unicast.hosts: ["10.0.0.30","10.0.0.74:9200"]
thread_pool.bulk.queue_size: 2000
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs

(Tim Vernum) #6

You've specified port 9200 in the ping hosts. That is the HTTP port, but Zen uses the transport port (9300) for discovery.


(Rahul Nama) #7

should I modify to

discovery.zen.ping.unicast.hosts: ["10.0.0.30","10.0.0.74"]
transport.tcp.port: 9300


(andy_zhou) #8

use ip not 9200
change to ["10.0.0.30","10.0.0.74"] and try


(Rahul Nama) #9

Hey @zqc0512

I've tried but no use. :frowning:

by any chance, do you think existing data is causing the problem?


(Christian Dahlqvist) #10

If you have 2 master eligible nodes, you should set minimum_master_nodes to 2 as well according to these guidelines.

Why are you setting this so high?


(Rahul Nama) #11

@Christian_Dahlqvist

I will set to 2 and give it a try.

regarding queue size, I've got bulk rejection exception, so I increased it.

Any dis-advantage If we keep it so high?


(Christian Dahlqvist) #12

Read this blog post on the topic. Increasing this parameter that much will add additional memory pressure without necessarily increasing throughput. How many indices/shards are you actively indexing into?


(andy_zhou) #13

you have two data node, one index have 2 replica?


(Rahul Nama) #14

@Christian_Dahlqvist

You can see it here

{"cluster_name":"my-cluster","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":114,"active_shards":114,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":107,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":51.58371040723983}

Also, If possible please let me know, having these many shards is a good practise or not?

by the way, thanks for the blog post


(Christian Dahlqvist) #15

Read this blog post for guidance on shards and sharding.


(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.