# Writing data from Spark to Elasticsearch

**URL:** <https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 14, 2017, 3:38pm UTC](https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414 "2017-06-14T15:38:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jobyjohny](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@jobyjohny](https://discuss.elastic.co/u/jobyjohny)\
**Post date:** [June 14, 2017, 3:38pm UTC](https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414/1 "2017-06-14T15:38:26Z")

</div>

I am using spark2.0 and using the spark session variable .trying to write some data into elastic search cluster installed in my local machine.code snippet is given below

```
spark.conf.set("es.index.auto.create", "true")
spark.conf.set("es.nodes", "127.0.0.1")
spark.conf.set("es.port", "9200")
val test = spark.createDataset(Seq(1,2))
EsSpark.saveToEs(test,"abc/def")

```

Last line is throwing a compilation error .Can some one suggest what is the correct way of calling the saveToEs method?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [June 15, 2017, 2:37pm UTC](https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414/2 "2017-06-15T14:37:21Z")

</div>

You'll need to transform your Dataset to be compatible with the DataFrame type, which means it must have a schema, and be capable of being transformed into a Dataset of type [Row] to be consumed.

---

<div class="post-metadata">

**Author:** ![jobyjohny](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@jobyjohny](https://discuss.elastic.co/u/jobyjohny)\
**Post date:** [June 19, 2017, 3:54pm UTC](https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414/3 "2017-06-19T15:54:02Z")

</div>

thank you James.  
I was able to write using SaveToEs method . code is given below  
val stats = spark.sql("select ... from table group by col1")  
stats.saveToEs("abc/def")

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2017, 3:54pm UTC](https://discuss.elastic.co/t/writing-data-from-spark-to-elasticsearch/89414/4 "2017-07-17T15:54:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.