ILM not deleting index

Elastic Stack Elasticsearch
21

Hi @raymondmintz11

I feel like I am guessing here but it sounds like you will stop writing to that index / data stream? These whole concepts are kinda around time series data... continuous streams...

But with that in mind. You will always have a rollover index, no way around that If you want the previous index go through the phases it must rollover...

Then there is a "fix" so to speak that if data stops flowing that new index will just sit and not really start the "age countdown" until some data is written this was to solve some other issue, otherwise many empty indices could be created say you had rollover every day / hour but delete after 30 days ... you would have lots of empty indices.... until they were deleted.

So if you really want that one empty index to go away (it takes up ~227 bytes primary) you will need to delete it... but then if data starts flowing again it will break.

Not clear why that "boot strap / empty index" is an issue.

You can look at some of the other settings to see if you can get it to do what you want but you would need to know "a priori"

I have not played with these much, perhaps you can get it to do what you want, but again you will need to intervene

Index Level ILM Settings

22

Youre correct, the current setup will stop writing to the index when the day is over. Our ingestor will start writing to the newly dated index.

Since I will always have a rollover index as you mentioned above, and our main goal is to not interevene, Ill move the setup to a continuous stream (no more date postfixed indexes)

This leads me to an important question:
In the old setup, I have a mechanism (script) that will decide what the shard count should be based on the previous index size..

Question: is there any dynamic shard count setting we can use?

For example: I want to increase the shard count (without doing a reindex) because we're seeing more traffic.. Therefore it would be good for me to either increase existing shard count or next "rollover" will have the adjusted/increased shard count.

Thanks so much for the insight

23

No there is no "dynamic primary shard count" setting

Primary Shard count is determined and index creation time.

You could update the template with the new primary shard count before rollover then the new index will get that at rollover.

Or you could update it after rollover and then force another rollover ... then you would have a small index in between.

24

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.