# How to increase writing speed to an index using Spark ES **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [December 7, 2021, 11:09pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165 "2021-12-07T23:09:19Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 7, 2021, 11:09pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/1 "2021-12-07T23:09:19Z")
Hi, I am trying to write a DataFrame with 10k rows and 31 columns into an Elasticsearch index using spark. I am using "JavaEsSparkSQL.saveToEs" function. There are 12 concurrent tasks. I used the default "es.batch.size.byte" and "es.batch.size.entries". It will take several seconds to write. I want to get it down to less than 1s. Is the "JavaEsSparkSQL.saveToEs" function doing bulk insert? Are there any good practice for increasing the indexing speed? Thanks in advance. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2021, 6:16am UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/2 "2021-12-08T06:16:05Z")
Which version of Elasticsearch are you using? What is the size and hardware specification of the cluster you are indexing into? How many indices and shards are you actively indexing into? Are you indexing immutable documents or also performing updates? ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [December 8, 2021, 7:14am UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/3 "2021-12-08T07:14:10Z")
Refer this as well : [Tune for indexing speed | Elasticsearch Guide [master] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html) ---
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 8, 2021, 2:59pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/4 "2021-12-08T14:59:42Z")
I am using 7.9.0 version. I currently have a single node cluster with Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz, 2592 Mhz, 6 Core(s), 12 Logical Processor(s), 32GB RAM. I am actively writing into one index with 5 primary shards and 5 replica shards. The rate will be a DataFrame with 10k and 31 columns in every 2 seconds. That means the index is getting updated every 2 seconds. Hope this information helps. I believe the data is not very large, the speed should be faster. It is just I am not familiar with how to tune it. Thanks for the help. ---
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 8, 2021, 3:03pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/5 "2021-12-08T15:03:13Z")
Thanks. I will look into this guide. However, I tried tuning "es.batch.size.byte" and "es.batch.size.entries" for bulk insert, which didn't make much difference. Do you know if .saveToEs using Bulk request? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2021, 3:36pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/6 "2021-12-08T15:36:26Z")
I would recommend upgrading Elasticsearch as I believe version 7.9.0 had a memory leak in Lucene. Elasticsearch is often limited by storage performance rather than CPU, especially when indexing. Are you using local SSDs? ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [December 8, 2021, 4:52pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/7 "2021-12-08T16:52:47Z")
Yes, saveToEs uses bulk requests controlled by the properties you mention. ---
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 8, 2021, 6:19pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/8 "2021-12-08T18:19:25Z")
Thanks for the reply. I am just using 1T local HDD. Does 7.10.2 version had the same problem as well? I will try with 7.10.2 and see the performance. So the indexing is bounded by the server's I/O performance? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2021, 7:32pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/9 "2021-12-08T19:32:37Z")
Elasticsearch 7.10 does not have that issue. > [@ElasticQuestion\_1234](#): > > So the indexing is bounded by the server's I/O performance? That is often the case, especially when slow storage is used. ---
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 8, 2021, 7:35pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/10 "2021-12-08T19:35:59Z")
Thanks, I will give 7.10 a try. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2021, 7:38pm UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/11 "2021-12-08T19:38:58Z")
Unless your total data volume dictates that you require 5 primary shards it might be worthwhile testing indexing into an index with a single primary shard. ---
**Author:** ![ElasticQuestion\_1234](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@ElasticQuestion\_1234](https://discuss.elastic.co/u/ElasticQuestion_1234)\ **Post date:** [December 9, 2021, 5:43am UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/12 "2021-12-09T05:43:16Z")
Thanks. I will try reducing primary shards as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 6, 2022, 5:43am UTC](https://discuss.elastic.co/t/how-to-increase-writing-speed-to-an-index-using-spark-es/291165/13 "2022-01-06T05:43:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.