Elastic index is bigger than expected

Elastic Search
1

We have an ILM policy which states that an index should roll over after 50GB

However our indices are all in excess of 300GB

This doesn't make sense...

2

How many primary and replica shards does the index have? ILM rollover size is generally based on the primary shard size and not the total size of the index, so it would help to also share the definition of your ILM policy.

3

Hi @Christian_Dahlqvist ,

I am unable to upload the screenshots for some reason it keeps failing.. I will do a text version here:

Enable rollover is set to true
Maximum primary shard size is not set
Maximum age is 1 day
Maximum documents is not set
Maximum index size is 50GB

Our total index size for one index is 326GB, but they all seem to be around that size. We have 102GB of primaries, and 3 shards, unassigned shards is 0.

We have an alert under stack monitoring saying our average shard size is 55GB.

index                                   prirep     shard    store
my-index-0.2024.08.01-100029            p          0        96

Note elastic version is 7.17.13

4

I think I figured it out. We are injecting data into the cluster using an elastic java client. It is manually creating indices and writing to them all simultaneously. I think this will invalidate the ILM.

Thanks for your reply @Christian_Dahlqvist

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.