# Tuning for high update and delete rate **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2013, 10:10pm UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079 "2013-10-23T22:10:07Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Dan\_Everton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_everton/32/1865_2.png) [@Dan\_Everton](https://discuss.elastic.co/u/Dan_Everton)\ **Post date:** [October 23, 2013, 10:10pm UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/1 "2013-10-23T22:10:07Z")
Howdy, One of our indices has a fairly high document count (around 100 million) and a high update and delete rate. In any given day, maybe a quarter of those documents will be updated, replaced, or deleted. Given this, the default merge settings don't seem to be keeping up and the percentage of deleted documents slowly creeps up. A lot of these documents are child documents so this seems to also increase memory pressure as the parent ID cache is growing along with the delete count. It only seems to clear when a whole segment is deleted. I've looked in to the merge and store settings and I'm wondering what the best course of action to address this is? So far I've increased the reclaim\_delete\_weight from 2.0 to 3.0 and changed max\_merge\_at\_once and segments\_per\_tier to 5. Store throttle is still enabled, but is set to 80mb/s and the hardware should be capable of at least that. I'm not completely sure that's the right direction as they don't really seem to have made any difference. Increasing the throttle rate doesn't seem to have reduced the store throttle time metric, and the deleted document ratio is still increasing. Anyone else had any experience with this sort of indexing load? In case I've made a typo somewhere, our actual settings look like this: indices.store.throttle.max\_bytes\_per\_sec: 80mb index.merge.policy.reclaim\_deletes\_weight: 3.0 index.merge.policy.max\_merge\_at\_once: 5 index.merge.policy.segments\_per\_tier: 5 Cheers, Dan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [October 25, 2013, 7:27am UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/2 "2013-10-25T07:27:46Z")
Hey, not sure, where your problem now is exactly. Do you have file system troubles, or is your main problem the memory consumption of the parent child cache? First, you should go with the latest elasticsearch version, parent child is constantly improved. Second you could add more nodes to prevent memory issues with parent child. Do you monitor your system and see the parent/child cache growing? If so, what are the numbers? Also, what is a high update/delete rate in numbers? The merge policy settings also have a couple of special settings regarding deletes, maybe you can play around with them, see > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. --Alex On Thu, Oct 24, 2013 at 12:10 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > Howdy, > > One of our indices has a fairly high document count (around 100 million) > and a high update and delete rate. In any given day, maybe a quarter of > those documents will be updated, replaced, or deleted. Given this, the > default merge settings don't seem to be keeping up and the percentage of > deleted documents slowly creeps up. A lot of these documents are child > documents so this seems to also increase memory pressure as the parent ID > cache is growing along with the delete count. It only seems to clear when a > whole segment is deleted. > > I've looked in to the merge and store settings and I'm wondering what the > best course of action to address this is? So far I've increased the > reclaim\_delete\_weight from 2.0 to 3.0 and changed max\_merge\_at\_once and > segments\_per\_tier to 5. Store throttle is still enabled, but is set to > 80mb/s and the hardware should be capable of at least that. I'm not > completely sure that's the right direction as they don't really seem to > have made any difference. Increasing the throttle rate doesn't seem to have > reduced the store throttle time metric, and the deleted document ratio is > still increasing. > > Anyone else had any experience with this sort of indexing load? In case > I've made a typo somewhere, our actual settings look like this: > > indices.store.throttle.max\_bytes\_per\_sec: 80mb > index.merge.policy.reclaim\_deletes\_weight: 3.0 > index.merge.policy.max\_merge\_at\_once: 5 > index.merge.policy.segments\_per\_tier: 5 > > Cheers, > Dan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dan\_Everton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_everton/32/1865_2.png) [@Dan\_Everton](https://discuss.elastic.co/u/Dan_Everton)\ **Post date:** [October 27, 2013, 10:45pm UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/3 "2013-10-27T22:45:06Z")
I guess the biggest concern for me is that the deleted document percentage always creeps up. The secondary effect is that the heap usage is strongly correlated as well because of the parent ID cache. I was hoping that by tweaking the merge settings we'd be able to get Elasticsearch to be more aggressive about expunging deletes and keep the deleted percentage closer to 10%. Right now we have to periodically do optimize calls just to keep it below 50%. So I'm hoping the reclaim deletes weight setting should be enough. We're also going to be testing disabling the store throttle. We're still on 0.90.2 but have an upgrade to 0.90.5 scheduled soon. Because of the ID cache stats bug in 0.90.2 we don't really have a good idea of what's going on with the ID cache. As for rates, we're averaging about 75 updates per second of which 50 of those will be delete operations. The update rate is very spiky though (user driven) so can go as high as 3000 updates per second (so about 2000 deletes per second). The majority of the document being changed are small child documents of much larger parent documents. The index has about 25 million documents. Cheers, Dan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [October 28, 2013, 12:45pm UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/4 "2013-10-28T12:45:54Z")
Dan, What do your performance metrics say is going on? Is your disk pegged? CPU? Historical metrics with something like Ganglia are best but you can get disk utilization with iostate -dmx 3 100 and get a very good idea of what is going on right now. Nik On Sun, Oct 27, 2013 at 6:45 PM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > I guess the biggest concern for me is that the deleted document percentage > always creeps up. The secondary effect is that the heap usage is strongly > correlated as well because of the parent ID cache. I was hoping that by > tweaking the merge settings we'd be able to get Elasticsearch to be more > aggressive about expunging deletes and keep the deleted percentage closer > to 10%. Right now we have to periodically do optimize calls just to keep it > below 50%. So I'm hoping the reclaim deletes weight setting should be > enough. We're also going to be testing disabling the store throttle. > > We're still on 0.90.2 but have an upgrade to 0.90.5 scheduled soon. > Because of the ID cache stats bug in 0.90.2 we don't really have a good > idea of what's going on with the ID cache. > > As for rates, we're averaging about 75 updates per second of which 50 of > those will be delete operations. The update rate is very spiky though (user > driven) so can go as high as 3000 updates per second (so about 2000 deletes > per second). The majority of the document being changed are small child > documents of much larger parent documents. The index has about 25 million > documents. > > Cheers, > Dan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dan\_Everton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_everton/32/1865_2.png) [@Dan\_Everton](https://discuss.elastic.co/u/Dan_Everton)\ **Post date:** [December 11, 2013, 11:15pm UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/5 "2013-12-11T23:15:34Z")
Just to follow this up. We've managed to sort out our settings to cope with the deletion rate we have. The key seemed to be disabling the store throttle completely. Elasticsearch is running on physical instances with fast(ish) disks so the throttle just seemed to be getting in the way. The final settings were indices.store.throttle.type: none index.merge.policy.reclaim\_deletes\_weight: 3.0 index.merge.policy.max\_merge\_at\_once: 5 index.merge.policy.segments\_per\_tier: 5 With those in place, the deleted document percent hovers around 15% rather than shooting up to 45% like it used to. Cheers, Dan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87cd5fa0-1af3-4bbe-8565-5fbc921d3c2e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/87cd5fa0-1af3-4bbe-8565-5fbc921d3c2e%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 12, 2013, 5:07am UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/6 "2013-12-12T05:07:36Z")
Good to know, thanks Dan. I think you could have also ... and still can increase index.merge.policy.reclaim\_deletes\_weight a bit more to push that 15% down a little further. ## Otis Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Wednesday, December 11, 2013 6:15:34 PM UTC-5, Dan Everton wrote: > Just to follow this up. We've managed to sort out our settings to cope > with the deletion rate we have. The key seemed to be disabling the store > throttle completely. Elasticsearch is running on physical instances with > fast(ish) disks so the throttle just seemed to be getting in the way. The > final settings were > > indices.store.throttle.type: none > index.merge.policy.reclaim\_deletes\_weight: 3.0 > index.merge.policy.max\_merge\_at\_once: 5 > index.merge.policy.segments\_per\_tier: 5 > > With those in place, the deleted document percent hovers around 15% rather > than shooting up to 45% like it used to. > > Cheers, > Dan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec9d60ad-5a32-4496-9ec4-d4db20ebaa31%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec9d60ad-5a32-4496-9ec4-d4db20ebaa31%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:01am UTC](https://discuss.elastic.co/t/tuning-for-high-update-and-delete-rate/14079/7 "2017-07-06T02:01:46Z")