No route to ElasticSearch from Spark in Standalone mode

[mtrivolli (Marco Trivolli)]

I'm working on some simple tests with Spark (standalone mode) and ElasticSearch. All my tests work fine as long as ElasticSearch is running in the same host as Spark. If I move ElasticSearch to a different node, then I get an exception "org.elasticsearch.hadoop.rest.EsHadoopTransportException: java.net.NoRouteToHostException: ..."

I have discarded any network / firewall issues, since I'm able to connect using kibana to the remote ElasticSearch node.

The test is as simple as this:

val spark = SparkSession.builder.master("local").appName("testing")
   .config("es.nodes", "remotenode")
   .getOrCreate()

spark.sparkContext.esRDD("radio/artists")
Console.out.println(RDD.count())

I'm using Spark 2.11 with elasticSearch-spark-20_2.11-5.3.2.

Any help?... I don't know what else to try!

[jkuang (Jimmy Kuang)]

On your remote ES node run the following:

nslookup remotenode

nslookup ON_IP_FROM_ABOVE

what is the output?

[mtrivolli (Marco Trivolli)]

nslookup:
Server: 80.58.61.250
Address: 80.58.61.250#53

** server can't find remotenode: NXDOMAIN```

nslookup:
Non-authoritative answer:
250.61.58.80.in-addr.arpa name = 250.red-80-58-61.staticip.rima-tde.net.

Authoritative answers can be found from:
61.58.80.in-addr.arpa nameserver = rsdmno1-14.rima-tde.net.
61.58.80.in-addr.arpa nameserver = rsdbgi1-11.rima-tde.net.

I don't have a DNS configured for my test network... Shall I?

[jkuang (Jimmy Kuang)]

Yes, you should configure DNS if you plan to use hostnames as the problem is it can not resolve what is "es.nodes" = "remotenode" . The alternative is you can always use the IP address , e.g. "es.nodes" = "192.68.1.101:9200"