ElasticSeach cluster - Error while adding new node to cluster

Elastic Stack Elasticsearch
1

Hi Team,

I am working on containerizing existing elastic cluster and I am planning to have setup like 2 server with multiple containers of elasticsearch & kibana within those servers using docker.

I am attaching design & compose files ( for server1 & server2) for creating different types of nodes and settings. So far I am able to create a 5 containers on server1 (being one for kibana) and I am trying to add few more nodes (as containers) to the existing cluster from different server. I am getting below error during adding of node ( from server2) .

Note: On server1 HOST = HOSTA, HOST2=HOSTB
On Server2 HOST=HOSTB, HOST2=HOSTA

Below are my docker-compose file snippets

Server-1:
services:
ess_elasticsearch1:
container_name: ess_elasticsearch1
environment:

  • cluster.name="ess_elasticsearch_cluster"
  • bootstrap.memory_lock=true
  • "ES_JAVA_OPTS=-Xms2g -Xmx2g"
  • "node.name=ess_elasticsearch1_${HOST}"
  • "network.bind_host=0.0.0.0"
  • "network.publish_host=${HOST}"
  • "discovery.zen.ping.unicast.hosts=${HOST}:9300, ${HOST}:9320, ${HOST}:9330, ${HOST}:9340"
  • "transport.host=${HOST}"
  • "transport.tcp.port=9300"
  • "index.number_of_shards: 5"
  • "index.number_of_replicas: 1"
  • "node.master=true"
  • "node.data=false"
  • "node.ingest=true"
    image: "docker.elastic.co/elasticsearch/elasticsearch:6.2.4"
    ports:
  • "9200:9200"
  • "9300:9300"
    ulimits:
    memlock:
    hard: -1
    soft: -1
    volumes:
  • "/usr/share/elasticsearch/data:/home/mwciuser/mw_modernization/elastic/data"
    network_mode: host
    ess_elasticsearch2:
    container_name: ess_elasticsearch2
    environment:
  • cluster.name="ess_elasticsearch_cluster"
  • bootstrap.memory_lock=true
  • "ES_JAVA_OPTS=-Xms2g -Xmx2g"
  • "ES_CONNECT_RETRY=30"
  • "node.name=ess_elasticsearch2_${HOST}"
  • "network.bind_host=0.0.0.0"
  • "network.publish_host=${HOST}"
  • "discovery.zen.ping.unicast.hosts=${HOST}:9300, ${HOST}:9320, ${HOST}:9330, ${HOST}:9340"
  • "transport.host=${HOST}"
  • "transport.tcp.port=9320"
  • "node.master=false"
  • "node.data=true"
  • "node.ingest=false"
    image: "docker.elastic.co/elasticsearch/elasticsearch:6.2.4"
    ports:
  • "9200:9220"
  • "9300:9320"
    ulimits:
    memlock:
    hard: -1
    soft: -1
    volumes:
  • "/usr/share/elasticsearch/data:/home/mwciuser/mw_modernization/elastic/data"
    network_mode: host

Server-2
services:
ess_elasticsearch3:
container_name: ess_elasticsearch3
environment:

  • cluster.name="ess_elasticsearch_cluster"
  • bootstrap.memory_lock=true
  • "ES_JAVA_OPTS=-Xms2g -Xmx2g"
  • "ES_CONNECT_RETRY=60"
  • "node.name=ess_elasticsearch3_${HOST}"
  • "network.bind_host=0.0.0.0"
  • "network.publish_host=${HOST}"
  • "discovery.zen.ping.unicast.hosts=${HOST2}:9300, ${HOST2}:9320, ${HOST2}:9330, ${HOST2}:9340"
  • "transport.host=${HOST}"
  • "transport.tcp.port=9330"
  • "node.master=true"
  • "node.data=false"
  • "node.ingest=true"
    image: "docker.elastic.co/elasticsearch/elasticsearch:6.2.4"
    ports:
  • "9200:9230"
  • "9300:9330"
    ulimits:
    memlock:
    hard: -1
    soft: -1
    volumes:
  • "/usr/share/elasticsearch/data:/home/mwciuser/mw_modernization/elastic/data"
    network_mode: host
    version: "2.2"

I was able to add 4 elasticsearch nodes of all containers sitting on same server, when I am trying to add 5th node from different server I am getting below error, Please see if anyone can help me on this?

[ess_elasticsearch3_labvlnx20.xxx.xxx.com] failed to send join request to master [{ess_elasticsearch4_labvlnx130.xxx.xxx.com}{6N6MMkndQWKO3PgxVsuqig}{4ELng_YgTrOz_W9TPGEEyQ}{labvlnx130.xxx.xxx.com}{10.64.4.xxx:9340}{ml.machine_memory=33569316864, ml.max_open_jobs=20, ml.enabled=true}], reason [RemoteTransportException[[ess_elasticsearch4_labvlnx130.xxx.xxx.com][10.64.4.xxx:9340][internal:discovery/zen/join]]; nested: ConnectTransportException[[ess_elasticsearch3_labvlnx20.xxx.xxx.com][10.64.4.xxx:9330] connect_exception]; nested: IOException[No route to host: labvlnx20.xxx.xxx.com/10.64.4.xxx:9330]; nested: IOException[No route to host]; ]

here is server-1 : labvlnx130.xxx.xxx.com is server-1 with 4 node working cluster
server-2 : labvlnx20.xxx.xxx.com is server-2 with a node adding and giving above error

I am able ping from both the servers each other on given ports.

I have gone through some of the issues available and tried setting like elasticsearch_network_host, - "transport.type=netty3" also but they did not help me

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.