# How to disable auto discovery in elasticsearch.haddop **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [November 20, 2015, 1:41pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149 "2015-11-20T13:41:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![vindoq](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vindoq](https://discuss.elastic.co/u/vindoq)\ **Post date:** [November 20, 2015, 1:41pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/1 "2015-11-20T13:41:53Z")
Hi, I'm running an elasticsearch client in a Scala project and doing some tests in a local environnement with a proxy (setup in $httpProxy, and my local machines are in the -Dhttp.nonProxyHosts java opts since they are local). I have configured my connector with this: ``` val client = ElasticClient.remote(settings, uri) ``` and only set the clusterName. but when I run my integration test, i got a: ``` ERROR [11-20-2015 14:03:41,478] [XCI=] org.apache.spark.executor.Executor - Exception in task 3.0 in stage 1.0 (TID 7) org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [GET] on [_nodes/transport] failed; server[null] returned [407|Proxy Authentication Required: [... this is a 407 proxy html page ...] at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:335) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:300) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:304) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:118) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:100) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:58) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:371) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:38) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEsWithMeta$1.apply(EsSpark.scala:87) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEsWithMeta$1.apply(EsSpark.scala:87) ~[elasticsearch-spark_2.10-2.1.0.Beta4.jar:2.1.0.Beta4] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) ~[spark-core_2.10-1.4.0.jar:1.4.0] at org.apache.spark.scheduler.Task.run(Task.scala:70) ~[spark-core_2.10-1.4.0.jar:1.4.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) [spark-core_2.10-1.4.0.jar:1.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80] at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80] TRACE [11-20-2015 14:03:41,480] [XCI=] org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport - Rx [...] TRACE [11-20-2015 14:03:41,480] [XCI=] org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport - Closing HTTP transport to localhost:9200 TRACE [11-20-2015 14:03:41,480] [XCI=] org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport - Closing HTTP transport to localhost:9200 ``` I'm pretty sure it's because the client want to discover other nodes but cannot reach them because it is behind a proxy. What options am I missing? where should I add them? Thanks. ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [November 23, 2015, 9:40pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/2 "2015-11-23T21:40:46Z")
Looks like [`es.nodes.wan.only`](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#_network) available in 2.2 ---
**Author:** ![vindoq](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vindoq](https://discuss.elastic.co/u/vindoq)\ **Post date:** [November 25, 2015, 1:15pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/3 "2015-11-25T13:15:30Z")
The problem is that it was working before when running a local InteliJ spark. So I should be able to solve it without upgrading. (perhaps...) I'm now seing what could be wrong: I have set one http client (ElasticClient.remote from sksamuel.elastic4s) to do some query and codes, and one spark client (ESSpark.saveToEsWithMetadata). Both have their configuration, here are the one for org.elasticsearch.spark: es.net.proxy.http.use.system.props to false es.nodes.discovery to false es.nodes.client.only to true es.noes to the client (I need to put a list, didn't have time to look that up for now). Before the error, when it was working with a local spark, I only used the http client from sksamuel.elastic4s. So it might just be a configuration error and maybe a conflict with the libraries? but what I find strange, it's org.elasticsearch.spark configuration doesn't configure the cluster name at all. I didn't find anything at all on either [https://github.com/elastic/elasticsearch-hadoop](https://github.com/elastic/elasticsearch-hadoop) or [https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html) is that normal? I'm trying to solve the configuration problem for the moment. ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [November 25, 2015, 9:22pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/4 "2015-11-25T21:22:41Z")
ES-Hadoop/Spark relies on REST and thus it doesn't need the cluster name, that is required for clients using the transport/java/node client (which effectively means the client becomes part of the cluster in one form or the other). ---
**Author:** ![vindoq](https://avatars.discourse-cdn.com/v4/letter/v/87869e/32.png) [@vindoq](https://discuss.elastic.co/u/vindoq)\ **Post date:** [December 2, 2015, 2:38pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/5 "2015-12-02T14:38:06Z")
Thanks to your answer, I manage to understand that the port was wrong, since it's via REST, so I put back 9200. Then I found someone with the same problem: [https://groups.google.com/forum/#!topic/elasticsearch/RQGz3DdmY24](https://groups.google.com/forum/#!topic/elasticsearch/RQGz3DdmY24) and had the same result: re-setting node discovery to true fixed my problem. I have no idea why, and the things I wanted is to disable it since we will have another elasticsearch cluster for logging the other application on the same network. Should I be worried, or is it standard and it would not affect the two different cluster since I specified all the nodes under the es.nodes conf? ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 8, 2015, 2:20pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/6 "2015-12-08T14:20:48Z")
I'm not sure what you are asking. You are confusing client routing with cluster discovery. I recommend stepping back a bit and picking up the reference documentation, in particular the getting started chapter. Don't rush going through it - it will not take too long and will explain the concepts behind ES architecture. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/how-to-disable-auto-discovery-in-elasticsearch-haddop/35149/7 "2017-07-06T13:26:58Z")