# Re-indexing past day-wise indices to reduce no of shards **URL:** \ **Category:** Elasticsearch\ **Created:** [December 6, 2018, 4:03am UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648 "2018-12-06T04:03:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 6, 2018, 4:03am UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/1 "2018-12-06T04:03:27Z")
In our ES 5.x cluster, we've 5 shards and 1 replica and we use day-wise indices with a retention of 150 days (150 indices at any time). The `pri.store.size` for a day-wise index is around 17 GB and with 1 replica the total index size per day is around 34 GB. With `5` shards and `pri.store.size` of `17 GB`, per shard it comes to be `3.5 GB` - that's very small shards and not efficient. Since we have day-wise indices, we could easily change the template and reduce the number of shards from `5` to `3` or even `1` without having the need to re-index. **Problem 1** : If i reduce the no of shards, I'm afraid my indexing performance might suffer. This cluster is mostly for storing metrics and at times during onboarding, a large influx of data happens. **Problem 2** : If I keep the no of shards as 5, my dashboard performance suffers since a query that spans 30 days is likely to hit at least 150 shards. What would be your opinion on the following approach: 1. Create a job that will run daily and `re-index` the previous day's index and change the no of shards to 1. This way, except for today's index which will have 5 shards, all the other indices will have 1 shard. 2. Create an alias that points to all indices `except` today's index and use that alias in dashboard / visualisations queries. In my use-case, it's okay if the current day's index data doesn't figure in. We are more interested in last 7 days data. 3. Create a job that will update the alias definitions daily. Our retention is 150 days. So the index that's purge needs to be removed from alias and on rollover, the previous day's index needs to be added in the alias. With this approach, I can let the indexing happen with 5 shards and the search happen with 1 shard per index since they would be re-index and aliased. OR am i better off changing the indices from daily to weekly? With that, a dashboard for 7 days would hit only 5 shards as against 35 currently. But I might lose the benefit of caching in this case, With daily indices, except today's index, the rest can be cached but with weekly indices only last week's index can be cached. I'd appreciate some inputs here and the best way to go about this. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 6, 2018, 8:03am UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/2 "2018-12-06T08:03:26Z")
There are two additional options you should consider. One is to use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-shrink-index.html) to reduce the primary shard count once the index is no longer being indexed into. The other is to use the [rollover index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-rollover-index.html) to create new indices as you reach a certain size rather that at a fixed interval. This allows you to get close to an ideal shard size and have indices cover varying time periods depending on how much data is coming in. It adds a lot of flexibility and is described in [this blog post](https://www.elastic.co/blog/managing-time-based-indices-efficiently). ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 6, 2018, 6:01pm UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/3 "2018-12-06T18:01:59Z")
Thank you Christian. I totally forgot about the `shrink` api inspite of having used it once before. I thought that once I re-index, I'll run a `_forcemerge` on all indices to reduce the underlying segments to 1. I think even if i do a `force merge` directly on the current RO indices without re-indexing, it should be fine. Am i correct here? I remember reading somewhere that `forcemerge` is way better than `shrink`. I did read on the `rollover` api but it's something we would want to do as a very last resort. Right now, we don't want to go with `rollover`. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 6, 2018, 6:43pm UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/4 "2018-12-06T18:43:25Z")
> [@sandeepkanabar](#): > > I remember reading somewhere that `forcemerge` is way better than `shrink` . They do different things and can be used together. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 6, 2018, 8:38pm UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/5 "2018-12-06T20:38:09Z")
Thanks Christian. So for the point `1` in the original post, I suppose I can create a job that calls the shrink api on previous day's index and then calls `forcemerge` on it for still better search efficiency? And what about point `2` and `3` ? Any suggestions? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2019, 8:38pm UTC](https://discuss.elastic.co/t/re-indexing-past-day-wise-indices-to-reduce-no-of-shards/159648/6 "2019-01-03T20:38:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.