# Compressing and forcemerging past time-based indices **URL:** \ **Category:** Elasticsearch\ **Created:** [November 20, 2020, 4:39am UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060 "2020-11-20T04:39:12Z")\ **Posts on this page:** 10\ **Page:** 2
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [November 25, 2020, 6:00pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/21 "2020-11-25T18:00:03Z")
Thanks @DavidTurner. You mean to say one force-merge thread for an entire cluster? Or per shard? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 25, 2020, 8:24pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/22 "2020-11-25T20:24:32Z")
That's a very good question, sorry I should have been more precise. It's one thread per node. So it's worth starting as many force-merges as you can as they'll run in sequence on each node, but in parallel across the whole cluster. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [November 25, 2020, 9:11pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/23 "2020-11-25T21:11:42Z")
Excellent. Thanks David for the clarification. Yeah I would usually run forcemerge using curator tool that is invoked via cronjob on each data node. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 9, 2020, 4:44pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/24 "2020-12-09T16:44:31Z")
Hi @Christian_Dahlqvist and @warkolm Just wanted to update you about some amazing performance numbers I ended up getting due to applying `best_compression` on past Read-only monthly indices and later force-merging them. Of the past 18 monthly indices that I compressed and force-merged, the minimum size reduction I achieved was a staggering `42.5%` (an index of 515 GB came down to 297 GB) with the maximum being `45%` (from 645 GB to 354 GB) . This has almost halved my storage requirements 🎉 Heartfelt thanks ! However, I have one question. When the monthly index was 645 GB initially (the monthly index was created by re-indexing day-wise indices into monthly to improve query performance), I had opted for 12 shards so that each shard will have around ~40-55 GB in size. But now with best\_compression, the size is almost down by 45% to 354 GB . My question is: **should I reduce the number of shards from 12 to 6?** Considering that these are already force merged, will shrinking from 12 to 6 help? Or that after shrinking, I've to again forcemerge? ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 10, 2020, 7:18pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/25 "2020-12-10T19:18:55Z")
Hey @Christian_Dahlqvist / @warkolm / @DavidTurner - will really appreciate some insights on this. Apologies for multiple tags. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 7:51pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/26 "2020-12-10T19:51:40Z")
Tricky to say. 20-30GB isn't unreasonably small for shards so IMO you could just leave them at 12; OTOH there is usually some data duplicated across shards (e.g. the terms dictionary) so shrinking might save you some more space. You don't _have_ to force-merge anything, and again it's tricky to say whether it'll improve things further. I don't think you'll find another 40% of space savings but it depends on the details of your data. Sorry there's no definitive answer here, I don't think we can offer more guidance than to try it and see. You've certainly got space to experiment now 🙂 ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 10, 2020, 8:09pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/27 "2020-12-10T20:09:28Z")
Thanks a ton, David. Totally understand. That helps a lot. I was just looking for some guidance that is it _worth trying or not_. From your inputs, it seems it would be worthwhile to try shrinking and see. I agree that 20-30 GB isn't too small for shards and that there may not be much impact by shrinking. But the only reason I thought to shrink was - when searching past 18 months data (current month index + past 18 monthly indices), the query currently hits 19 indices \* 12 shards = 228 shards. I thought that if I reduce to 6, the query will hit just 114 shards. That was my reasoning. Of course, not expecting `40%` space savings again. More than happy with the gains I've obtained 😅 So I suppose, I can run curator to `shrink` shards from `12` to `6`. And since the data is already force merged, I reckon each index will end up with 6 segments for primary instead of current 12. Thanks for all your inputs. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2020, 8:27pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/28 "2020-12-10T20:27:07Z")
> [@sandeepkanabar](#): > > I thought that if I reduce to 6, the query will hit just 114 shards. That makes sense; searching fewer larger shards can be more efficient (because of how the data structures scale) but it can also be less efficient (e.g. uses fewer parallel threads). It all depends... > [@sandeepkanabar](#): > > And since the data is already force merged, I reckon each index will end up with 6 segments for primary instead of current 12. Shrinking doesn't necessarily adjust the segment count. If you currently have twelve 20GB shards each with a single segment then shrinking by a factor of two will almost certainly leave you with six 40GB shards each with **two** segments. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 10, 2020, 8:32pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/29 "2020-12-10T20:32:45Z")
> [@DavidTurner](#): > > Shrinking doesn't necessarily adjust the segment count. If you currently have twelve 20GB shards each with a single segment then shrinking by a factor of two will almost certainly leave you with six 40GB shards each with **two** segments. And that means, after shrinking, it will need to be force-merged if I want to reduce the segments to 1? > [@DavidTurner](#): > > That makes sense; searching fewer larger shards can be more efficient (because of how the data structures scale) but it can also be less efficient (e.g. uses fewer parallel threads). It all depends... Got it. In that case, I think I'm better off with what I've now 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 7, 2021, 8:32pm UTC](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060/30 "2021-01-07T20:32:52Z")
Closed on January 7, 2021, 8:32pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/compressing-and-forcemerging-past-time-based-indices/256060.md?page=1)