# saveToEs Write performance (elasticsearch-spark)

**URL:** https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [July 7, 2016, 1:27pm UTC](https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947 "2016-07-07T13:27:27Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![larghir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larghir/32/61526_2.png) [@larghir](https://discuss.elastic.co/u/larghir)
#### Post date: [July 7, 2016, 1:27pm UTC](https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947/1 "2016-07-07T13:27:27Z")

</div>

Hi,

I have a question about parallelism when saving an RDD to Elasticsearch.  
I have an RDD (created with SparkSQL) with 1000 partitions, and an Elasticsearch index with 5 primary shards. I run my application on a Spark cluster with 3 executors.  
However, I only see one task (running on one executor) when calling saveToEs, though I would expect it to write in parallel.  
What is going wrong there?

---

<div class="post-metadata">

### Author: ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)
#### Post date: [September 15, 2016, 4:22pm UTC](https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947/2 "2016-09-15T16:22:08Z")

</div>

Did you ever get to the bottom of this issue? I am seeing the same thing

---

<div class="post-metadata">

### Author: ![larghir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larghir/32/61526_2.png) [@larghir](https://discuss.elastic.co/u/larghir)
#### Post date: [September 22, 2016, 11:25am UTC](https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947/3 "2016-09-22T11:25:05Z")

</div>

Hi @Pat_Humphreys,

see this answer:

> [@Performance degradation when writing to AWS elasticsearch using elasticsearch-hadoop library](https://discuss.elastic.co/t/performance-degradation-when-writing-to-aws-elasticsearch-using-elasticsearch-hadoop-library/50298/6):
>
> @larghir The situation with keys being shuffled to one reducer is primarily a MapReduce case. A Spark RDD will write out to Elasticsearch in parallel using which ever number of partitions are configured. Writing parallelism does also depend on your RDD layout, your configuration, and the available resources in your environment.

I ended up using saveToEsWithMeta, configuration including at least values for es.batch.size.entries, es.batch.size.bytes, es.batch.write.retry.count.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/savetoes-write-performance-elasticsearch-spark/54947/4 "2017-07-06T13:23:09Z")

</div>


