ILM policy is not followed correctly and index is not getting rollover as per doc size or max age

Elastic Stack Kibana
1

Hello All,

I'm facing major challenges with regards to ILM policy for all my indexes.

In ILM policy I'm using two phase HOT and DELETE.
Hot Phase- It should have max 2mb of data or data should be there for 3 min then rollover.
Issue faced- The data keep on writing on same index, i.e became 5mb and after some time it got rollover i.e it didn't even follow 3min defined in policy.

Why dosent policy is followed strictly like if more than 2MB or more than 3 min either it should have rolled over.But it takes some random time to get rolled over in HOT face.

Delete phase- Defined to delete data after 2 seconds from rollover,even though now index was rollover as per above condition but didnt got deleted after 2 seconds.

What if the data become to large then this delay will also increase?,this is a major issue.

Ideally my data can be from 1GB to 30 Gb or 500Mb as well and want to delete this data after 1 month and new data should be rolled over to new index.But above is the challenge faced.

Any suggestion would be helpful.

image
image1452×301 28 KB 

PUT _ilm/policy/cis-metric-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "3m",
            "max_size": "2mb"
          },
          "set_priority": {
            "priority": null
          }
        }
      },
      "delete": {
        "min_age": "2s",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Thanx

2

don't know why it is happening
may be too short max_age?

try putting like 30m and max_size like 1gig. see if it rolls over.
I have 50gig and 2 days and it does rollover around 49.x or 50.x gig everyday.

1 Like
3

Hi @PRASHANT_MEHTA Those sizes and timeframes are too short..

ILM is not really design to operate on "Minute" Scheduled or Small KB/MB indices is designed to manage large scale indices GB, Hours, Days etc. and for that it works very well.

I posted a bit about it here and another elastician here

2 Likes
4

Hello @stephenb
Can you please tell me , what if I want to rollover my indices on monthly basis.
I want to generate rollover indices for SEPT...OCT...NOV...DEC
I know we can use "max_age":"30d" for that , but it will create problem for the months having 31 days & 28 days in a year.

 "actions": {
          "rollover": {
            "max_age": "(Monthly)",
5

what is the reason for having this?
are you thinking you can drop them at your convivence?

if you want index by Month you will have to define index name in logstash something like
index_name-MM

6

"Calendar Units" are not currently supported as far as I know with ILM Hours, Days, GB etc. I do not think they are on the road map.

Most users think of ILM Policies in Days... 7 days Hot 23 Days Warm 30 Days Cold etc... regardless of the Calendar month.

1 Like
7

Yah, Even I also thought the same.
But wanted to confirm. Thank you so much !

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.