# Bulk update is too slow elasticsearch 6.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2018, 5:01am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213 "2018-04-24T05:01:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 5:01am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/1 "2018-04-24T05:01:50Z")
Hi guys, Hers is my configuration: ES version = 6.2 JVM = 30gb Ram = 128gb CPU = 24core SDK = PHP I am experiencing Bulk update is too slow. I tried [given solution](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-update-settings.html) which not working. Any suggestion will be appreciable Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 5:40am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/2 "2018-04-24T05:40:21Z")
How are you updating the documents? What performance are you seeing? What is the size and structure of your documents? How frequently is an individual document updated? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 5:46am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/3 "2018-04-24T05:46:58Z")
I am doing bulk updating of 500 documents. Per document size is 32 kb approx. I am having queue system. So i wrote daemon which continuous pop the queue in batch of 500 and updating to ES. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 5:49am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/4 "2018-04-24T05:49:34Z")
How many parallel update threads are you running? Are you using nested documents or parent-child? How many updates do you get per second? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 5:54am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/5 "2018-04-24T05:54:34Z")
20 php child process is running. There is some nested doc and main doc too which being update. I am also using script. Mostly updates are using 'doc\_as\_upsert'. If the batch size is 50 its hitting 4 bulk api request/sec. Which means 50\*4 = 200 doc updating / sec. I would like to update approx 5k/persec. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 5:56am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/6 "2018-04-24T05:56:59Z")
What does CPU usage, disk I/O and iowait look like on the Elasticsearch host? How many indices and shards are you actively updating? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 6:02am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/7 "2018-04-24T06:02:32Z")
with 'iostat' its showing: Linux 3.10.0-693.11.1.el7.centos.plus.x86\_64 avg-cpu: %user %nice %system %iowait %steal %idle 2.71 0.48 0.43 0.02 0.00 96.37 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 6:18am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/8 "2018-04-24T06:18:57Z")
> [@Christian\_Dahlqvist](#): > > How frequently is an individual document updated? > [@Christian\_Dahlqvist](#): > > How many indices and shards are you actively updating? Do you see anything in the Elasticsearch logs, e.g. around slow GC or merging falling behind? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 6:23am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/9 "2018-04-24T06:23:09Z")
How frequently is an individual document updated? =\> two update every 2 sec/doc How many indices and shards are you actively updating? =\> 30 indices with 10 shards, 1replica I am not seeing any log related to gc and merging. Should i increase the threads ? as the cpu idle is 96%. Or is their any other conf? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 6:29am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/10 "2018-04-24T06:29:03Z")
If you update a document that is still present in the transaction log and has not yet been written to a segment, this will trigger a refresh. If you're frequently updating the same documents, this will hurt performance as a refresh is an expensive operation. How many nodes do you have in the cluster? How much data do you have in total? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 6:32am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/11 "2018-04-24T06:32:27Z")
Yes i tried by disable/increase/decrease refresh\_internal = 0/-1/30 but not got any success. I am having single node/cluster architecture . Right now i am having approx 10 million data which keep increasing. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 6:39am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/12 "2018-04-24T06:39:58Z")
As you are updating documents quite frequently, I do not necessarily think a longer refresh interval will help at all. It may actually be better to leave it at the default 1 second. ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 6:41am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/13 "2018-04-24T06:41:44Z")
hmm .. okay So by adding node OR increasing threads will solve this problem ? OR is their any imp conf which i am missing ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 24, 2018, 6:43am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/14 "2018-04-24T06:43:54Z")
I do not have a lot of experience of update intensive use cases, so am not sure how to best optimise this. Let's see if someone else chimes in. ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 24, 2018, 12:16pm UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/15 "2018-04-24T12:16:10Z")
Hi i just noticed in my log In which i got 20 child process simultaneously send 4 bulk request/sec and 1 process also send 4 bulk req/sec. I think ES is not handling multiple connection for bulk . Is their any conf/setting for bulk api ? ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [April 24, 2018, 5:26pm UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/16 "2018-04-24T17:26:25Z")
Man, I do, and it wasn't a good experience! 😉 If you are updating the same document multiple times per second, and you have many threads of execution updating documents, you will bury Elasticsearch. Occasional bursts of updates: fine. Constant low-frequency updates: fine. Constant high-frequency updates: not fine. And it gets less fine the larger the doc size. I would suggest using a different approach to handle the frequent updates. Perhaps an in-memory cache that periodically (i.e., once every minute or two) updates a record in ES. ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 25, 2018, 4:48am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/17 "2018-04-25T04:48:02Z")
Thanks for reply loren. So is their any way where i can update in realtime. I have 2k to 3k per sec. How i can handle this in realtime OR it is not good practice to push data in realtime into elasticsearch ? ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [April 25, 2018, 5:29am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/18 "2018-04-25T05:29:17Z")
You can index tons of data into Elasticsearch very rapidly. Just not rapid updates, in my experience. ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [April 26, 2018, 4:19pm UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/19 "2018-04-26T16:19:55Z")
Hi i guys, I debug and found the cause. I was heavily using script condition as well as update with upsert. So i just remove script condition and kept update query with upsert & got performance. But still m facing version control by running multiple threads . Any idea guys how i can tackle this ? ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [May 7, 2018, 7:56am UTC](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213/20 "2018-05-07T07:56:07Z")
Hi guys, I got solution which helped me to increase my performance . You can check here [https://gist.github.com/ashishtiwari1993/004a19f4a44efc214403a7fc1ee27cda#challenge-1-](https://gist.github.com/ashishtiwari1993/004a19f4a44efc214403a7fc1ee27cda#challenge-1-) [Next page](https://discuss.elastic.co/t/bulk-update-is-too-slow-elasticsearch-6-2/129213.md?page=2)