# ES - Spark tuning for bulk writes **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [December 19, 2020, 4:22pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167 "2020-12-19T16:22:42Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 19, 2020, 4:22pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/1 "2020-12-19T16:22:42Z")
I;m using Spark 3 with ES 7.10 , ECK 1.3 I have following settings ```auto |es.batch.size.bytes|6000000| |es.batch.size.entries|10000| |es.batch.write.refresh|false| |es.batch.write.retry.count|6| ``` Would like to know how can we tune properly. Whats the base criteria. I have tried to ```auto "transient": { "indices.store.throttle.type": "none" }, "persistent" : { "threadpool.bulk.type": "fixed", "threadpool.bulk.size": 60, "threadpool.bulk.queue_size": 3000, "threadpool.generic.keep_alive": "5m" } }' ``` But its not happening with ES 7.10. How can we measure the data written per request at elasticsearch and identify the bottleneck . ES i3 2xlarge - 5 data nodes, 3 master nodes. Also rejection write thread has high number, Output from one node ```auto "write": { "threads": 6, "queue": 0, "active": 0, "rejected": 481, "largest": 6, "completed": 625803 } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2020, 7:17pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/2 "2020-12-19T19:17:19Z")
How many indices and shards are you actively indexing into? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 20, 2020, 4:10pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/3 "2020-12-20T16:10:19Z")
> [@Christian\_Dahlqvist](#): > > ards are you actively indexing into? Indexes around 4 per day, with 24 shards in each ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2020, 4:17pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/4 "2020-12-20T16:17:29Z")
Why so many shards? How much data do you index per day? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 20, 2020, 4:49pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/5 "2020-12-20T16:49:39Z")
> [@Christian\_Dahlqvist](#): > > ow much data do you index per day? Around 900m per day. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2020, 5:07pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/6 "2020-12-20T17:07:48Z")
How much is that in GB? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 20, 2020, 5:35pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/7 "2020-12-20T17:35:40Z")
Its approx min ~850GB to max 1000GB per index per day ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2020, 5:42pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/8 "2020-12-20T17:42:02Z")
The best way to improve efficiency and [reduce the risk of rejection problems](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster) is often to limit the number of shards you index into. Instead of having daily indices with a large number of shards I would recommend switching to using rollover and ILM. That way you can set the number of primary shards to 5 per index (same as your number of data nodes) and have rollover switch to new underlying indices based on size (often recommended to around 50GB per shard) and/or time. In your case it probably means that you would generate multiple indices per day where each cover a shorter time period. This should be more efficient. ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 20, 2020, 5:48pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/9 "2020-12-20T17:48:12Z")
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: es\_rejected\_execution\_exception: rejected execution of org.elasticsearch.ingest.IngestService$3@1f629a72 on EsThreadPoolExecutor[name = ct-es-es-data-nodes-0/write, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4e705b38[Running, pool size = 6, active threads = 6, queued tasks = 622, completed tasks = 547409]] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 20, 2020, 5:53pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/10 "2020-12-20T17:53:46Z")
Also make sure you are following [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/tune-for-indexing-speed.html) if you are not already. ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 20, 2020, 6:17pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/11 "2020-12-20T18:17:52Z")
yes @Christian_Dahlqvist, read this already and applied them. ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 21, 2020, 3:39pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/12 "2020-12-21T15:39:38Z")
@Christian_Dahlqvist, still jobs are taking time, and my data nodes cpu are less than 30% only. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 21, 2020, 3:46pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/13 "2020-12-21T15:46:42Z")
Indexing tends to be disk I/O intensive, so CPU is often not the limiting factor. As you are using `i3` instance I suspect you should be fine though. How many concurrent clients/connections are you using for indexing? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 22, 2020, 3:02am UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/14 "2020-12-22T03:02:21Z")
Yes im using i3 instance with provision IOPS . Using currently 32 connections ```auto org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: es_rejected_execution_exception: rejected execution of coordinating operation [coordinating_and_primary_bytes=68231502, replica_bytes=44867860, all_bytes=113099362, coordinating_operation_bytes=6528502, max_coordinating_and_primary_bytes=107374182] ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 22, 2020, 8:16am UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/15 "2020-12-22T08:16:54Z")
Are you sending bulk requests to all data nodes, avoiding the master nodes? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 23, 2020, 9:36am UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/16 "2020-12-23T09:36:36Z")
@Christian_Dahlqvist I have deployed using ECK. So was using http service created by that eck deployment kibana-kb-http ClusterIP 172.20.102.107 5601/TCP 3d21h es-es-data-nodes ClusterIP None 27h es-es-http ClusterIP 172.20.153.65 9200/TCP 27h es-es-master ClusterIP None 27h using es-es-http service which has 9200 port exposed. Master doesnt have same port exposed. ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [December 27, 2020, 3:32pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/17 "2020-12-27T15:32:05Z")
@Christian_Dahlqvist , i have following from one of the node iostat ```auto avg-cpu: %user %nice %system %iowait %steal %idle 38.51 0.00 2.44 0.55 0.06 58.43 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn xvda 1.37 1.38 15.62 345772 3915036 nvme0n1 0.00 0.01 0.00 2138 0 xvdc 370.29 397.94 25726.29 99759173 6449231588 ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 24, 2021, 3:32pm UTC](https://discuss.elastic.co/t/es-spark-tuning-for-bulk-writes/259167/18 "2021-01-24T15:32:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.