Re-indexing past day-wise indices to reduce no of shards

[sandeepkanabar]

In our ES 5.x cluster, we've 5 shards and 1 replica and we use day-wise indices with a retention of 150 days (150 indices at any time).

The pri.store.size for a day-wise index is around 17 GB and with 1 replica the total index size per day is around 34 GB. With 5 shards and pri.store.size of 17 GB, per shard it comes to be 3.5 GB - that's very small shards and not efficient.

Since we have day-wise indices, we could easily change the template and reduce the number of shards from 5 to 3 or even 1 without having the need to re-index.

Problem 1: If i reduce the no of shards, I'm afraid my indexing performance might suffer. This cluster is mostly for storing metrics and at times during onboarding, a large influx of data happens.

Problem 2: If I keep the no of shards as 5, my dashboard performance suffers since a query that spans 30 days is likely to hit at least 150 shards.

What would be your opinion on the following approach:

1. Create a job that will run daily and re-index the previous day's index and change the no of shards to 1. This way, except for today's index which will have 5 shards, all the other indices will have 1 shard.

2. Create an alias that points to all indices except today's index and use that alias in dashboard / visualisations queries. In my use-case, it's okay if the current day's index data doesn't figure in. We are more interested in last 7 days data.

3. Create a job that will update the alias definitions daily. Our retention is 150 days. So the index that's purge needs to be removed from alias and on rollover, the previous day's index needs to be added in the alias.

With this approach, I can let the indexing happen with 5 shards and the search happen with 1 shard per index since they would be re-index and aliased.

OR

am i better off changing the indices from daily to weekly? With that, a dashboard for 7 days would hit only 5 shards as against 35 currently. But I might lose the benefit of caching in this case, With daily indices, except today's index, the rest can be cached but with weekly indices only last week's index can be cached.

I'd appreciate some inputs here and the best way to go about this.

[Christian_Dahlqvist]

There are two additional options you should consider.

One is to use the shrink index API to reduce the primary shard count once the index is no longer being indexed into.

The other is to use the rollover index API to create new indices as you reach a certain size rather that at a fixed interval. This allows you to get close to an ideal shard size and have indices cover varying time periods depending on how much data is coming in. It adds a lot of flexibility and is described in this blog post.

[sandeepkanabar]

Thank you Christian. I totally forgot about the shrink api inspite of having used it once before. I thought that once I re-index, I'll run a _forcemerge on all indices to reduce the underlying segments to 1. I think even if i do a force merge directly on the current RO indices without re-indexing, it should be fine. Am i correct here?

I remember reading somewhere that forcemerge is way better than shrink.

I did read on the rollover api but it's something we would want to do as a very last resort. Right now, we don't want to go with rollover.

[Christian_Dahlqvist]

@sandeepkanabar

I remember reading somewhere that forcemerge is way better than shrink .

They do different things and can be used together.

[sandeepkanabar]

Thanks Christian. So for the point 1 in the original post, I suppose I can create a job that calls the shrink api on previous day's index and then calls forcemerge on it for still better search efficiency?

And what about point 2 and 3 ? Any suggestions?