Elasticsearch7.4 multinode cluster - Bootstrap Service discovery on Openshift cluster

Elastic Stack Elasticsearch
1

Hi,

I am working on upgrading 3 node cluster elasticsearch5.5 hosted on openshift to elasticsearch7.4. Everything works on dev/qa ( single node cluster).
My solution is to create brand new cluster and then migrate historical dataset and live dataset into it.

While going through the process of multi node cluster (2 nodes) in UAT, I have come across an issue. Bootstrap configuration requires hostname/IP addresses as static configuration. This is an issue as Openshift creates pods on fly hence assigns IP/hostname dynamically, so it is hard to predict host IP/hostname to configure.

I made pods as StatefulSets to make hostname predictable (2 node cluster):

  • elasticsearch7-0
  • elasticsearch7-1

However IP can not be predicted.
Configuartion:

cluster.initial_master_nodes: ["elasticsearch7-0", "elasticsearch7-1"]
discovery.seed_hosts: ["elasticsearch7-0", "elasticsearch7-1"]

node2 config:
cluster.initial_master_nodes: ["elasticsearch7-0", "elasticsearch7-1"]
discovery.seed_hosts: ["elasticsearch7-0", "elasticsearch7-1"]

Node names of each name are set to hostname:
node.name: ${HOSTNAME}

The discovery fails
{"type": "server", "timestamp": "2020-01-30T11:45:08,416Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "eai-es", "node.name": "elasticsearch7-1", "message": "failed to resolve host [elasticsearch7-0]",
"stacktrace": ["java.net.UnknownHostException: elasticsearch7-0: Name or service not known",
"at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]",
"at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930) ~[?:?]",
"at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1499) ~[?:?]",
"at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:849) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]",

I wonder if anyone has used elasticsearch7+ on Openshift, this would be a major for everyone. I do not see any warning in elasticsearch7 documentation clearly stating issues around Kub/Openshift deploy.

I will look forward to any pointers in this regard.

Regards
Jay

2

Your approach sounds like a good one - the master-eligible nodes need predictable names and a StatefulSet is a good way to achieve that.

This means that the name elasticsearch7-0 can't be resolved to an IP address using whatever name resolver is available to this node. Are you sure this is a valid DNS name?

3

This looks like it might be related:

Is your service headless? It probably should be. Every Elasticsearch node does its own routing of incoming requests, so I think there's no need to use a ClusterIP.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.