I have submitted a bug ticket on github, but I got redirected here. Here is the ticket.
In short I found that the 'discovery.zen.ping.unicast.host' could resolve IP addresses of nodes in my cluster via discovery service name, but 'discovery.seed_hosts' could only resolve them if the static IP of the discovery service were given.
The problem here:
Documentation lacks this information
Hard coding IP of discovery service could lead to problems as the cluster naturally scales and redistributes between nodes.
I find this a regression, a feature loss therefore a bug. What are you opinions?
These docs specifically say that discovery.seed_hosts will do DNS lookups:
The host is either a host name to be resolved by DNS, an IPv4 address, or an IPv6 address. IPv6 addresses must be enclosed in square brackets. If a host name resolves via DNS to multiple addresses, Elasticsearch uses all of them. DNS lookups are subject to JVM DNS caching.
Moreover there really is no difference between the handling of discovery.zen.ping.unicast.host and discovery.seed_hosts, they both feed into the configuredHosts variable in the following code and from then on there is no distinction:
I think the difference in behaviour you're seeing is most likely environmental.
unfortunatelly we already eliminated all environmental changes. It is literally only the elasticsearch image version and this discovery setting as differences.
Elasticsearch 7.17.0 has a feature which allows to do this and 8.8.1 does not have it under this setting. I tried to dig into the java code, but I'm no expert in that.
Question: how can one handle a cluster restart, scaling, redistribution or restarting of nodes?