# o.e.d.z.UnicastZenPing - failed to resolve host: java.net.UnknownHostException

**URL:** <https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507>\
**Category:** Elasticsearch\
**Created:** [March 29, 2019, 9:47am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507 "2019-03-29T09:47:00Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 9:47am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/1 "2019-03-29T09:47:00Z")

</div>

Hello. I have a simple problem. My elasticsearch nodes don't communicate each other. Could you help me? 😟

Elasticsearch 6.7.0(installed from rpm)  
CentOS Linux 7.6.1810  
Linux 3.10.0-957.5.1.el7.x86\_64 on x86\_64

I did these:

1. configured servers to use LAN IP pool(10.0.0.X/24) and stopped firewalls on all nodes
2. configured .yml file on all nodes
3. cross-checked with "curl 10.0.0.x:9200" and got successful results
4. cross-checked nslookup with "nslookup esX" and got successful results

Here is the log of elasticsearch:

> [2019-03-29T12:31:01,652][INFO][o.e.e.NodeEnvironment] [es1] using [1] data paths, mounts [[/ (rootfs)]], net usable\_space [671.1gb], net total\_space [899.5gb], types [rootfs]  
> [2019-03-29T12:31:01,655][INFO][o.e.e.NodeEnvironment] [es1] heap size [989.8mb], compressed ordinary object pointers [true]  
> [2019-03-29T12:31:01,683][INFO][o.e.n.Node] [es1] node name [es1], node ID [-JW1C2VySX6IQGQcBWQTXg]  
> [2019-03-29T12:31:01,684][INFO][o.e.n.Node] [es1] version[6.7.0], pid[18024], build[default/rpm/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/3.10.0-957.5.1.el7.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_201/25.201-b09]  
> [2019-03-29T12:31:01,684][INFO][o.e.n.Node] [es1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5039937088361580624, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs\_err\_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [aggs-matrix-stats]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [analysis-common]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [ingest-common]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [ingest-geoip]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [ingest-user-agent]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [lang-expression]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [lang-mustache]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [lang-painless]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [mapper-extras]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [parent-join]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [percolator]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [rank-eval]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [reindex]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [repository-url]  
> [2019-03-29T12:31:03,194][INFO][o.e.p.PluginsService] [es1] loaded module [transport-netty4]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [tribe]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-ccr]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-core]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-deprecation]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-graph]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-ilm]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-logstash]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-ml]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-monitoring]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-rollup]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-security]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-sql]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-upgrade]  
> [2019-03-29T12:31:03,195][INFO][o.e.p.PluginsService] [es1] loaded module [x-pack-watcher]  
> [2019-03-29T12:31:03,196][INFO][o.e.p.PluginsService] [es1] no plugins loaded  
> [2019-03-29T12:31:06,447][INFO][o.e.x.s.a.s.FileRolesStore] [es1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]  
> [2019-03-29T12:31:06,995][INFO][o.e.x.m.p.l.CppLogMessageHandler] [es1] [controller/18145] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV  
> [2019-03-29T12:31:07,355][DEBUG][o.e.a.ActionModule] [es1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security  
> [2019-03-29T12:31:07,598][INFO][o.e.d.DiscoveryModule] [es1] using discovery type [zen] and host providers [settings]  
> [2019-03-29T12:31:08,255][INFO][o.e.n.Node] [es1] initialized  
> [2019-03-29T12:31:08,255][INFO][o.e.n.Node] [es1] starting ...  
> [2019-03-29T12:31:08,362][INFO][o.e.t.TransportService] [es1] publish\_address {10.0.0.1:9300}, bound\_addresses {10.0.0.1:9300}  
> [2019-03-29T12:31:08,385][INFO][o.e.b.BootstrapChecks] [es1] bound or publishing to a non-loopback address, enforcing bootstrap checks

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 9:47am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/2 "2019-03-29T09:47:36Z")

</div>

> [2019-03-29T12:31:08,433][WARN][o.e.d.z.UnicastZenPing] [es1] failed to resolve host [“es1”]  
> java.net.UnknownHostException: “es1”: Temporary failure in name resolution  
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0\_201]  
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0\_201]  
> at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:737) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0\_201]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]  
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]  
> [2019-03-29T12:31:08,438][WARN][o.e.d.z.UnicastZenPing] [es1] failed to resolve host [“es2”]  
> java.net.UnknownHostException: “es2”: Temporary failure in name resolution  
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0\_201]  
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0\_201]  
> at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:737) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0\_201]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]  
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]  
> [2019-03-29T12:31:11,443][WARN][o.e.d.z.ZenDiscovery] [es1] not enough master nodes discovered during pinging (found [[Candidate{node={es1}{-JW1C2VySX6IQGQcBWQTXg}{JaSkzeqjRsS5FDjuwwms\_Q}{10.0.0.1}{10.0.0.1:9300}{ml.machine\_memory=33535541248, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again

The results from cross-checking network:

> [root@dhpc01 ~]# ping 10.0.0.1 -c 1  
> PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.  
> 64 bytes from 10.0.0.1: icmp\_seq=1 ttl=64 time=0.040 ms
> 
> --- 10.0.0.1 ping statistics ---  
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms  
> rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms  
> [root@dhpc01 ~]# ping 10.0.0.2 -c 1  
> PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.  
> 64 bytes from 10.0.0.2: icmp\_seq=1 ttl=64 time=0.136 ms
> 
> --- 10.0.0.2 ping statistics ---  
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms  
> rtt min/avg/max/mdev = 0.136/0.136/0.136/0.000 ms  
> [root@dhpc01 ~]# ping es1 -c 1  
> PING es1 (10.0.0.1) 56(84) bytes of data.  
> 64 bytes from dhpc01 (10.0.0.1): icmp\_seq=1 ttl=64 time=0.042 ms
> 
> --- es1 ping statistics ---  
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms  
> rtt min/avg/max/mdev = 0.042/0.042/0.042/0.000 ms  
> [root@dhpc01 ~]# ping es2 -c 1  
> PING es2 (10.0.0.2) 56(84) bytes of data.  
> 64 bytes from dhpc02 (10.0.0.2): icmp\_seq=1 ttl=64 time=0.152 ms
> 
> --- es2 ping statistics ---  
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms  
> rtt min/avg/max/mdev = 0.152/0.152/0.152/0.000 ms

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 9:48am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/3 "2019-03-29T09:48:13Z")

</div>

> [root@dhpc01 ~]# curl es1:9200  
> {  
> "name" : "es1",  
> "cluster\_name" : "demir-elastic",  
> "cluster\_uuid" : "_na_",  
> "version" : {  
> "number" : "6.7.0",  
> "build\_flavor" : "default",  
> "build\_type" : "rpm",  
> "build\_hash" : "8453f77",  
> "build\_date" : "2019-03-21T15:32:29.844721Z",  
> "build\_snapshot" : false,  
> "lucene\_version" : "7.7.0",  
> "minimum\_wire\_compatibility\_version" : "5.6.0",  
> "minimum\_index\_compatibility\_version" : "5.0.0"  
> },  
> "tagline" : "You Know, for Search"  
> }  
> [root@dhpc01 ~]# curl es2:9200  
> {  
> "name" : "es2",  
> "cluster\_name" : "demir-elastic",  
> "cluster\_uuid" : "_na_",  
> "version" : {  
> "number" : "6.7.0",  
> "build\_flavor" : "default",  
> "build\_type" : "rpm",  
> "build\_hash" : "8453f77",  
> "build\_date" : "2019-03-21T15:32:29.844721Z",  
> "build\_snapshot" : false,  
> "lucene\_version" : "7.7.0",  
> "minimum\_wire\_compatibility\_version" : "5.6.0",  
> "minimum\_index\_compatibility\_version" : "5.0.0"  
> },  
> "tagline" : "You Know, for Search"  
> }  
> [root@dhpc01 ~]# nslookup es1  
> ;; Got SERVFAIL reply from 193.140.25.30, trying next server  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es1  
> Address: 10.0.0.1
> 
> [root@dhpc01 ~]# nslookup es2  
> ;; Got SERVFAIL reply from 193.140.25.30, trying next server  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es2  
> Address: 10.0.0.2

> [root@dhpc02 ~]# curl es1:9200  
> {  
> "name" : "es1",  
> "cluster\_name" : "demir-elastic",  
> "cluster\_uuid" : "_na_",  
> "version" : {  
> "number" : "6.7.0",  
> "build\_flavor" : "default",  
> "build\_type" : "rpm",  
> "build\_hash" : "8453f77",  
> "build\_date" : "2019-03-21T15:32:29.844721Z",  
> "build\_snapshot" : false,  
> "lucene\_version" : "7.7.0",  
> "minimum\_wire\_compatibility\_version" : "5.6.0",  
> "minimum\_index\_compatibility\_version" : "5.0.0"  
> },  
> "tagline" : "You Know, for Search"  
> }  
> [root@dhpc02 ~]# curl es2:9200  
> {  
> "name" : "es2",  
> "cluster\_name" : "demir-elastic",  
> "cluster\_uuid" : "_na_",  
> "version" : {  
> "number" : "6.7.0",  
> "build\_flavor" : "default",  
> "build\_type" : "rpm",  
> "build\_hash" : "8453f77",  
> "build\_date" : "2019-03-21T15:32:29.844721Z",  
> "build\_snapshot" : false,  
> "lucene\_version" : "7.7.0",  
> "minimum\_wire\_compatibility\_version" : "5.6.0",  
> "minimum\_index\_compatibility\_version" : "5.0.0"  
> },  
> "tagline" : "You Know, for Search"  
> }  
> [root@dhpc02 ~]# nslookup es1  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es1  
> Address: 10.0.0.1
> 
> [root@dhpc02 ~]# nslookup es2  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es2  
> Address: 10.0.0.2

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 9:48am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/4 "2019-03-29T09:48:46Z")

</div>

```
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: demir-elastic
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es1
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: [“es1” , “es2”, “es3” , “es4” , “es5” , “es6” , “es7” , “es8” , “es9” , “es10” , “es11” , “es12” , “es13” , “es14” , “es15” , “es16”]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 29, 2019, 10:15am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/5 "2019-03-29T10:15:43Z")

</div>

Thanks for the detailed report @lemon_soft.

> [@lemon\_soft](#):
>
> ```auto
> [root@dhpc01 ~]# nslookup es1
> ;; Got SERVFAIL reply from 193.140.25.30, trying next server
> Server: 127.0.0.1
> Address: 127.0.0.1#53
> 
> Name: es1
> Address: 10.0.0.1
> 
> [root@dhpc01 ~]# nslookup es2
> ;; Got SERVFAIL reply from 193.140.25.30, trying next server
> Server: 127.0.0.1
> Address: 127.0.0.1#53
> 
> Name: es2
> Address: 10.0.0.2
> 
> ```

I'm suspicious about those `SERVFAIL reply` messages. It looks like the first lookup is failing and then `nslookup` is retrying with a different DNS server, and I am wondering if the JVM's DNS resolver doesn't do this. Can you adjust your DNS config so that these lookups succeed on the first try?

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 10:28am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/6 "2019-03-29T10:28:46Z")

</div>

Thanks DavidTurner. I changed the order of DNS. Problem is still existing.

The reply from nslookup:

> [root@dhpc01 ~]# nslookup es1  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es1  
> Address: 10.0.0.1
> 
> [root@dhpc01 ~]# nslookup es2  
> Server: 127.0.0.1  
> Address: 127.0.0.1#53
> 
> Name: es2  
> Address: 10.0.0.2

The error log again:

> [2019-03-29T13:28:39,872][WARN][o.e.d.z.ZenDiscovery] [es1] not enough master nodes discovered during pinging (found [[Candidate{node={es1}{-JW1C2VySX6IQGQcBWQTXg}{7FRT-\_vmQQmc181c03qWiw}{10.0.0.1}{10.0.0.1:9300}{ml.machine\_memory=33535541248, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [3]), pinging again  
> [2019-03-29T13:28:39,873][WARN][o.e.d.z.UnicastZenPing] [es1] failed to resolve host [“es1”]  
> java.net.UnknownHostException: “es1”  
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0\_201]  
> at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:737) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0\_201]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]  
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]  
> [2019-03-29T13:28:39,874][WARN][o.e.d.z.UnicastZenPing] [es1] failed to resolve host [“es2”]  
> java.net.UnknownHostException: “es2”  
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0\_201]  
> at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0\_201]  
> at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:737) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0\_201]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201]  
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201]

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 29, 2019, 10:39am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/7 "2019-03-29T10:39:14Z")

</div>

Hmm. Ok, it's definitely something wrong with name lookups still:

> [@lemon\_soft](#):
>
> ```auto
> [2019-03-29T13:28:39,874][WARN][o.e.d.z.UnicastZenPing] [es1] failed to resolve host [“es2”]
> java.net.UnknownHostException: “es2”
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_201]
> 
> ```

Just to check, did you restart the node after fixing the DNS config? I think it only picks up changes when restarted.

I think my next step would be to investigate what's actually happening at the DNS level using `tcpdump`, comparing the successful query generated by `nslookup` to the failing one generated by Elasticsearch. Perhaps this'll give us a clue.

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 11:45am UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/8 "2019-03-29T11:45:59Z")

</div>

Yes, I restarted elasticsearch.

> Just to check, did you restart the node after fixing the DNS config? I think it only picks up changes when restarted.

> I think my next step would be to investigate what's actually happening at the DNS level using `tcpdump`, comparing the successful query generated by `nslookup` to the failing one generated by Elasticsearch. Perhaps this'll give us a clue.

I ran a tcpdump(tcpdump -i any udp port 53 \> tcpdump.txt). Here is the dump [file download link](http://localhost/tcpdump.txt)(uploaded for text length limit).

I should also tell about DNSMasq. I don't use DNS server for resolving hosts. In my environment, I am not authorized to add/update DNS server records. I use DNSMasq whish is using /etc/hosts file to respond DNS queries.

> [root@dhpc01 ~]# tcpdump -D  
> 1.nflog (Linux netfilter log (NFLOG) interface)  
> 2.nfqueue (Linux netfilter queue (NFQUEUE) interface)  
> 3.em1  
> 4.usbmon1 (USB bus number 1)  
> 5.em2  
> 6.usbmon2 (USB bus number 2)  
> 7.any (Pseudo-device that captures on all interfaces)  
> 8.lo [Loopback]

I noticed that a strange record in tcpdump.txt:

> [root@dhpc01 ~]# ping es2.cs.deu.edu.tr  
> ping: es2.cs.deu.edu.tr: Name or service not known  
> [root@dhpc01 ~]# ping es2  
> PING es2 (10.0.0.2) 56(84) bytes of data.  
> 64 bytes from dhpc02 (10.0.0.2): icmp\_seq=1 ttl=64 time=0.159 ms  
> 64 bytes from dhpc02 (10.0.0.2): icmp\_seq=2 ttl=64 time=0.180 ms  
> ^C  
> --- es2 ping statistics ---  
> 2 packets transmitted, 2 received, 0% packet loss, time 1000ms  
> rtt min/avg/max/mdev = 0.159/0.169/0.180/0.016 ms

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 29, 2019, 12:06pm UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/9 "2019-03-29T12:06:05Z")

</div>

> [@lemon\_soft](#):
>
> Here is the dump [file download link](http://demed.cs.deu.edu.tr/aliriza/tcpdump.txt)(uploaded for text length limit).

I can't see _any_ successful lookups over DNS in the log you shared.

The text output from `tcpdump` doesn't normally include enough information for diagnosis. In general it's better to look at the raw dump:

`tcpdump -i any udp port 53 -s65535 -wtcpdump.pcap`

Then open `tcpdump.pcap` in something like [Wireshark](https://www.wireshark.org).

However, I think I see the problem. The queries in the log look rather odd:

```auto
14:34:01.876591 IP dhpc01.cs.deu.edu.tr.27170 > tnz1030.tinaztepe.deu.edu.tr.domain: 40676+ A? M-bM-^@M-^\es2M-bM-^@M-^]. (27)

```

Here's what a normal query looks like on my laptop:

```auto
11:59:17.368076 IP 192.168.1.179.51532 > 192.168.1.12.53: 3345+ A? es1. (21)

```

Note all that extra junk `M-bM-^@M-^\` that shouldn't be there. I looked more closely at your config file and it seems you have so-called "smart" quotes there:

```auto
discovery.zen.ping.unicast.hosts: [“es1” , “es2”, “es3” , “es4” , “es5” , “es6” , “es7” , “es8” , “es9” , “es10” , “es11” , “es12” , “es13” , “es14” , “es15” , “es16”]

```

Compare this to a correctly-formatted file:

```auto
discovery.zen.ping.unicast.hosts: ["es1" , "es2", "es3" , "es4" , "es5" , "es6" , "es7" , "es8" , "es9" , "es10" , "es11" , "es12" , "es13" , "es14" , "es15" , "es16"]

```

Note the subtly different quotation marks. Try fixing that.

I also note that you have `discovery.zen.minimum_master_nodes: 3`. This is appropriate for clusters with 4 or 5 master-eligible nodes. How many of your nodes are master-eligible? Normally you'd only list the master-eligible nodes in `discovery.zen.ping.unicast.hosts`, but you have listed 16 there. If you have 16 master-eligible nodes then `discovery.zen.minimum_master_nodes` should be set to 9.

---

<div class="post-metadata">

**Author:** ![lemon\_soft](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemon_soft/32/43072_2.png) [@lemon\_soft](https://discuss.elastic.co/u/lemon_soft)\
**Post date:** [March 29, 2019, 12:18pm UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/10 "2019-03-29T12:18:04Z")

</div>

> [@DavidTurner](#):
>
> Note all that extra junk `M-bM-^@M-^\` that shouldn't be there. I looked more closely at your config file and it seems you have so-called "smart" quotes there:
> 
> > discovery.zen.ping.unicast.hosts: ["es1" , "es2", "es3" , "es4" , "es5" , "es6" , "es7" , "es8" , "es9" , "es10" , "es11" , "es12" , "es13" , "es14" , "es15" , "es16"]

**It worked after changing smart quotes! OMG!**

> Status: Green  
> Nodes: 16  
> Indices: 14  
> Memory: 4.8 GB / 15.5 GB  
> Total Shards: 25  
> Unassigned Shards: 0  
> Documents: 263,052  
> Data: 218.8 MB

Thank you so much @DavidTurner. You saved my week! Thanks.

I will dig this dump further:

> Note all that extra junk `M-bM-^@M-^\` that shouldn't be there.

Note:  
I am crying because of using Webmin interface to edit files. Webmin text edit causes "smart" quote problem! Also elasticsearch does care about that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2019, 12:18pm UTC](https://discuss.elastic.co/t/o-e-d-z-unicastzenping-failed-to-resolve-host-java-net-unknownhostexception/174507/11 "2019-04-26T12:18:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.