Index Lifecycle Management rolleover not happening properly

amiya-santra · October 12, 2020, 1:35pm

I have manually configured ILM.

After creating policies and telling my indices to use them, they all complain with errors like this:

issue: rollover index have 0 doc

PUT _ilm/policy/new_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "2h",
            "max_size": "100mb"
          },
          "set_priority": {
            "priority": 10
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Template

PUT _template/prod-log-temp
{
  "index_patterns": [
    "my-log-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "new_policy",
        "rollover_alias": "my-log"
      }
    }
  }
}

My policy doing rollover index but its containing 0 doc

first index: my-log-00001(only incrising size of this index)
second index: my-log-00002 (having 0 doc)

Can any one help why this is happening.

kavierkoo · October 12, 2020, 2:35pm

I'm suspecting your logstash conf's output for this specific index is not pointing to ILM alias,
Could you share the logstash .conf's output?

amiya-santra · October 13, 2020, 2:34am

Thanks for your quick response.
I am not using logstash for transferring log I am using filebeat, here is my filebeat output configuration.

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:13.xx.xx.xx}:${ELASTICSEARCH_PORT:31116}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      indices:
        - index: "%{[type]}-000001"
          when.equals:
                  type: "my-log"
    setup.ilm.enabled: false
    ilm.enabled: false

kavierkoo · October 13, 2020, 6:32am

As I was suspected, your index is pointing to %{[type]}-000001 hence the ilm not working.
You need to point to your alias instead of an index name with the 000001.

Some useful info:

Hope this could help!

amiya-santra · October 13, 2020, 8:47am

But my %{[type]} parameter value is ```
my-log

It means both are the same [ type] value and my alias.

so you mean to say don't need to provide index as 
index: "%{[type]}-000001" I can write like this
index: "%{[type]}".

kavierkoo · October 13, 2020, 9:06am

If your %{[type]} match your rollover-alias it should work.

But before you do the changes, do you mind to share below info for a better understanding?

GET /my-log-00001 (Just wanted to see the aliases portion)
GET /my-log-00002 (Just wanted to see the aliases portion)
GET my-log-00001/_ilm/ explain
GET my-log-00002/_ilm/ explain

amiya-santra · October 14, 2020, 5:54am

Now its creating 2 indexes after that its giving this error
illegal_argument_exception: index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]

GET prod-nmn-000001/_ilm/ explain

{
  "indices" : {
    "prod-nmn-000001" : {
      "index" : "prod-nmn-000001",
      "managed" : true,
      "policy" : "new_policy",
      "lifecycle_date_millis" : 1602650088714,
      "age" : "28.23m",
      "phase" : "hot",
      "phase_time_millis" : 1602650088802,
      "action" : "rollover",
      "action_time_millis" : 1602650621674,
      "step" : "ERROR",
      "step_time_millis" : 1602651221583,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]",
        "stack_trace" : """
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92)
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:274)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:175)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:203)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)

"""
      },
      "phase_execution" : {
        "policy" : "new_policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "100mb",
              "max_age" : "2h"
            },
            "set_priority" : {
              "priority" : 10
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1602499425529
      }
    }
  }
}

GET /prod-nmn-000001

{
  "prod-nmn-000001" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "flags" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "gw_timestamp" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "loglevel" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "messages" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "network-id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "new_policy",
          "rollover_alias" : "prod-nmn"
        },
        "number_of_shards" : "1",
        "provided_name" : "prod-nmn-000001",
        "creation_date" : "1602650088714",
        "priority" : "10",
        "number_of_replicas" : "0",
        "uuid" : "-E-yDN6hQFmRxpfl_gia0w",
        "version" : {
          "created" : "7040299"
        }
      }
    }
  }
}

kavierkoo · October 14, 2020, 6:31am

It shows alias is empty for this index.

Did you bootstrap your index after you created your policy and template?

ILM require below steps:

Create a lifecycle policy that defines the appropriate phases and actions. See Create a lifecycle policy above.
Create an index template to apply the policy to each new index.
Bootstrap an index as the initial write index.
Verify indices are moving through the lifecycle phases as expected.

also, isn't the index name my-log-xxxxx, why the log shows prod-nmn-xxxxxx instead?

amiya-santra · October 14, 2020, 7:00am

Thanks Kavier,
I changed the index name from my-log to prod-nmn. can we do it dynamically? because my index is going to create dynamically so how to do bootstrap for dynamically created index? as I check in example

PUT timeseries-000001
{
  "aliases": {
    "timeseries": {
      "is_write_index": true
    }
  }
}

they are bootstrapping only for this index timeseries-000001 what about other index which ae going to create after rollover?

kavierkoo · October 14, 2020, 7:16am

This is where the index template come in as you configured below.

you only have to bootstap the first one, and subsequently as long as the index pattern in your index template matched, it will auto assign the rollover alias to your xxx-000002 index

I strongly suggest to have a read up on the whole ILM document as it is really helpful to understand how ILM works.
Also I find Datasteam doc helps me understand ILM even better.

Hope this can help you!

amiya-santra · October 14, 2020, 11:19am

Thanks Kavier,

I went through the document it's help a lot to understand ILM . one small question I am giving rollover index size 100mb but my rollover index are containing more then 100 mb log like 350Mb,284Mb,456Mb. Why it's happening like this?

kavierkoo · October 16, 2020, 5:18am

Can you run another round of below commands to have a better idea:

GET INDEXNAME-000001/_ilm/ explain
Dry run following your existing policy for that specific index

amiya-santra · October 16, 2020, 7:44am

I applied the policy and template in a new
alias "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296"

GET 0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001/_ilm/explain

{
  "indices" : {
    "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001" : {
      "index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001",
      "managed" : true,
      "policy" : "new-device-policy",
      "lifecycle_date_millis" : 1602825221945,
      "age" : "2.38h",
      "phase" : "hot",
      "phase_time_millis" : 1602824303020,
      "action" : "complete",
      "action_time_millis" : 1602825222393,
      "step" : "complete",
      "step_time_millis" : 1602825222393,
      "phase_execution" : {
        "policy" : "new-device-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "1mb",
              "max_age" : "1h"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1602819713971
      }
    }
  }
}

I did a dry run on my alias with max age 1 day and max size 1 mb as i mention in policy.

POST /0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296/_rollover?dry_run
{
  "conditions" : {
    "max_age": "1d",
    "max_size": "1mb"
  }
}

Response

{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000009",
  "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010",
  "rolled_over" : false,
  "dry_run" : true,
  "conditions" : {
    "[max_age: 1d]" : false,
    "[max_size: 1mb]" : true
  }
}

where i am doing wrong?

kavierkoo · October 16, 2020, 7:59am

After this one did you see the index got rollover to xxx-000002?

amiya-santra · October 16, 2020, 8:09am

index are getting rollover by its own, now I have total 10 index after getting rollover and it keeps going. after doing a dry run in responce its printing old index and new index but logs are going to the old index, no new index is getting created immediately its happening by its own after some time.

{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010",
  "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000011",
  "rolled_over" : false,
  "dry_run" : true,
  "conditions" : {
    "[max_age: 1d]" : false,
    "[max_size: 1mb]" : true
  }
}

kavierkoo · October 16, 2020, 8:32am

Dry run is a simulation, it should not execute the rollover action.

Anyway, it seems like its working already isnt it?

I dont really get this part, could you elaborate on this?

amiya-santra · October 16, 2020, 8:39am

if it's doing a dry run then ```
acknowledged

yes, it's working but the size I have provided in policy to rollover (2mb) so  rollover index should container 2 mb log not more then that .. but my rollover index contain 25 to 60 mb logs.. why its happenig?

kavierkoo · October 16, 2020, 9:00am

There's an interval of 10 minutes that rollover will happen

indices.lifecycle.poll_interval
(Dynamic, time unit value) How often index lifecycle management checks for indices that meet policy criteria. Defaults to 10m .

Rollover will only run after every interval, if within the interval your logs keeps coming in, it will still store at the previous even after the policy hits.

system · November 13, 2020, 9:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I just want a simple ILM rollover policy Elasticsearch ilm-index-lifecycle-management	3	18312	May 2, 2019
Rollover ILM policy for existing Index Logstash ilm-index-lifecycle-management	6	402	March 1, 2022
Life Cycle Policies: Alias for rollover index Elasticsearch ilm-index-lifecycle-management	1	546	January 19, 2021
Help with simple ILM rollover configuration for existing index Elasticsearch ilm-index-lifecycle-management	2	202	August 17, 2023
Index rollover issues Elasticsearch ilm-index-lifecycle-management	8	1409	January 11, 2022

Index Lifecycle Management rolleover not happening properly

Related Topics