I'm facing major challenges with regards to ILM policy for all my indexes.
In ILM policy I'm using two phase HOT and DELETE.
Hot Phase- It should have max 2mb of data or data should be there for 3 min then rollover.
Issue faced- The data keep on writing on same index, i.e became 5mb and after some time it got rollover i.e it didn't even follow 3min defined in policy.
Why dosent policy is followed strictly like if more than 2MB or more than 3 min either it should have rolled over.But it takes some random time to get rolled over in HOT face.
Delete phase- Defined to delete data after 2 seconds from rollover,even though now index was rollover as per above condition but didnt got deleted after 2 seconds.
What if the data become to large then this delay will also increase?,this is a major issue.
Ideally my data can be from 1GB to 30 Gb or 500Mb as well and want to delete this data after 1 month and new data should be rolled over to new index.But above is the challenge faced.
Can you please tell me , what if I want to rollover my indices on monthly basis.
I want to generate rollover indices for SEPT...OCT...NOV...DEC
I know we can use "max_age":"30d" for that , but it will create problem for the months having 31 days & 28 days in a year.