# Saving dataframe to ElasticSearch

**URL:** <https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [October 21, 2019, 11:32am UTC](https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451 "2019-10-21T11:32:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hugh\_McBride](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hugh_mcbride/32/46843_2.png) [@Hugh\_McBride](https://discuss.elastic.co/u/Hugh_McBride)\
**Post date:** [October 21, 2019, 11:32am UTC](https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451/1 "2019-10-21T11:32:05Z")

</div>

I am on a new project that is saving spark dataframes to elastic search using

dataFrame.saveToEsI(.... )

There are multi terabytes of Data (\> 10) that are being saved to a rather small cluster (5 node ) ,  
And it is taking on the order of a week to load, Is this method of saving data to elastic search suitable for this volume of data, (I am guessing not ) or would it be better to save the dataframes to json ( the spark/hadoop cluster is much much bigger ) first and then bulkload

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [November 14, 2019, 7:23pm UTC](https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451/2 "2019-11-14T19:23:06Z")

</div>

The saving process already converts data into json and performs successive bulkloading to Elasticsearch, so doing this outside of the connector probably won't see any changes to write speed. Ten-plus terrabytes seems like a large amount of data for a 5 node cluster, though I'm not familiar with your deployment so it's hard for me to say it's unreasonable.

One thing that might make sense to take a look at is the [batch size settings](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#configuration-serialization) on ES-Hadoop. Increasing these may allow you to send larger bulk requests and cut down on high amounts of back and forth traffic.

Additionally, disabling refresh on your indices and in ES-Hadoop may help the ingestion rate in that Elasticsearch may spend less time flushing smaller segments to disk.

---

<div class="post-metadata">

**Author:** ![Hugh\_McBride](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hugh_mcbride/32/46843_2.png) [@Hugh\_McBride](https://discuss.elastic.co/u/Hugh_McBride)\
**Post date:** [November 15, 2019, 11:15am UTC](https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451/3 "2019-11-15T11:15:04Z")

</div>

James, I suspected the elastic cluster being a bit on the small side, thanks for your response.

vr  
hmb

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2019, 11:15am UTC](https://discuss.elastic.co/t/saving-dataframe-to-elasticsearch/204451/4 "2019-12-13T11:15:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.