Unable to join to my new cluster

Elastic Stack Elasticsearch
1

Hello,

We are very inexperienced with ES, so we imagine we are just overlooking something simple.

We have built two machines, running ES (OSS) 7.8.

  1. Hostname: g000057.domain.com, IP: 10.40.1.62, Node name: g000057
  2. Hostname: g000059.domain.com, IP: 10.40.1.192, Node name: g000059

g000057 works fine standalone, we have data coming in from logstash and filebeat and can view/search the data in Kibana.

We want to add server 2 for high availability and generally experimenting with.

The problem is that we see this is server 1's log:

[g000057] failed to validate incoming join request from node [{g000059}{p5NnwKJ7TRykGI2wHEUgoA}{rfQBBSF1Sj-48n4idaoIPA}{10.40.1.192}{10.40.1.192:9300}{dimr}]

And this is server 2's log:

[2020-07-08T10:51:27,730][INFO ][o.e.c.c.JoinHelper       ] [g000059] failed to join {g000057}{1KWcemjuQQW9u3Im7Ha7qQ}{bExQ55uESI65Bif8b_qlnA}{10.40.1.62}{1org.elasticsearch.transport.RemoteTransportException: [g000057][10.40.1.62:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node

Server 1's config:

cluster.name: "es-cluster"
node.name: "g000057"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.40.1.62
http.port: 9200
discovery.seed_hosts: 
        - 10.40.1.62
        - g000059.domain.com
cluster.initial_master_nodes: 
        - g000057

Server 2's config:

cluster.name: "es-cluster"
node.name: "g000059"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.40.1.192
http.port: 9200
discovery.seed_hosts:
    - g000057.domain.com
    - 10.40.1.192
cluster.initial_master_nodes: ["g000057"]

Can someone advise what's wrong with the configuration please?

Thank you!

2

Do you have IPv6 enabled but not configured? Try turning it off, that worked for me last time (ES 7.6). Don't know if it's a current issue anymore.

edit: to be more precise. My cloudprovider runs their networks by default using IPv4, which is fine for most usecases anyway. But this means their virtual networks don't support IPv6 unless specifically configured. However every linux distro has IPv6 enabled by default. So for some reason elasticsearch wanted to use IPv6, even if I specified network.host: 0.0.0.0

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.