# Multiple ES clusters in SparkSQL **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [November 25, 2015, 7:59am UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523 "2015-11-25T07:59:20Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ludonara](https://avatars.discourse-cdn.com/v4/letter/l/9dc877/32.png) [@ludonara](https://discuss.elastic.co/u/ludonara)\ **Post date:** [November 25, 2015, 7:59am UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/1 "2015-11-25T07:59:20Z")
Hi all, is there a way to handle multipe ES clusters in a Spark application? For example, use sparkSQL to query data from two ES clusters. For now, i've just seen configuration of "es.nodes" in sparkConf. Is it possible to specify "es.node" when read datasource? ex: sqlContext.read.format("org.elasticsearch.spark.sql")**.option("es.node","X.X.X.X")**.load("logstash-2015.10.30/type") Thank you. ---
**Author:** ![ludonara](https://avatars.discourse-cdn.com/v4/letter/l/9dc877/32.png) [@ludonara](https://discuss.elastic.co/u/ludonara)\ **Post date:** [November 25, 2015, 9:58am UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/2 "2015-11-25T09:58:21Z")
this is related to my question [Using ES Spark to copy data from one instance to another](https://discuss.elastic.co/t/using-es-spark-to-copy-data-from-one-instance-to-another/34568) and the response seems "no way for now" 😅 ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [November 25, 2015, 9:21pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/3 "2015-11-25T21:21:09Z")
I would rephrase this. Within the same `RDD`/`DataFrame` and thus within the _same_ Spark task you can't read and write data at the same time. You can however use a temporary storage to stream the data between them. ---
**Author:** ![ludonara](https://avatars.discourse-cdn.com/v4/letter/l/9dc877/32.png) [@ludonara](https://discuss.elastic.co/u/ludonara)\ **Post date:** [November 26, 2015, 7:17pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/4 "2015-11-26T19:17:55Z")
We would use Elasticsearch as datasource for a BI application. In this use case, it would be a nice feature if we could specify es.node IP via "read..option" (to handle multiple ES conf and switch beetween them on the fly). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [December 8, 2015, 2:08pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/5 "2015-12-08T14:08:25Z")
Have you tried using it? Any ES-Hadoop configuration can be passed per method call - the connector will merge them (last one wins) and run the job. ---
**Author:** ![ludonara](https://avatars.discourse-cdn.com/v4/letter/l/9dc877/32.png) [@ludonara](https://discuss.elastic.co/u/ludonara)\ **Post date:** [January 5, 2016, 1:12pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/6 "2016-01-05T13:12:45Z")
Hi, and happy new year. Indeed, That works! Thank you. We have just one other problem with array type and nested schema (as mentionned here : [Spark-sql does not seem to read from a nested schema](https://discuss.elastic.co/t/spark-sql-does-not-seem-to-read-from-a-nested-schema/24672/11)) to be fully compatible with Elasticsearch as datasource for our BI application. Best regards, Ludovic CEO @datarocksIO ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 9, 2016, 1:25pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/7 "2016-01-09T13:25:26Z")
This has been fixed in the latest ES Hadoop release, 2.2-rc1 as described [here](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/mapping.html#mapping-multi-values). Please try it out. ---
**Author:** ![ludonara](https://avatars.discourse-cdn.com/v4/letter/l/9dc877/32.png) [@ludonara](https://discuss.elastic.co/u/ludonara)\ **Post date:** [January 11, 2016, 7:42am UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/8 "2016-01-11T07:42:51Z")
That works fine for array of primitives types. But I get a `java.lang.NullPointerException` on a field which is typed as `NESTED` on mapping. Eg. of Mapping : `INFO ScalaEsRowRDD: Discovered mapping {index=[mappings=[dashboard=[doc_meta_id=STRING, id=STRING, language=STRING, sheets=NESTED, template=STRING, tenant_id=STRING, workbook_ids=STRING]]]} for [index/dashboard]` ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 11, 2016, 9:01am UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/9 "2016-01-11T09:01:03Z")
Can you post a simple snippet in Spark which creates the DataFrame and then reads it? JSON works just fine (and typically makes things easier). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/multiple-es-clusters-in-sparksql/35523/10 "2017-07-06T13:26:35Z")