Index Lifecycle Management rolleover not happening properly

Elastic Stack Kibana
1

I have manually configured ILM.

After creating policies and telling my indices to use them, they all complain with errors like this:

issue: rollover index have 0 doc

PUT _ilm/policy/new_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "2h",
            "max_size": "100mb"
          },
          "set_priority": {
            "priority": 10
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Template

PUT _template/prod-log-temp
{
  "index_patterns": [
    "my-log-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "new_policy",
        "rollover_alias": "my-log"
      }
    }
  }
}

My policy doing rollover index but its containing 0 doc

first index: my-log-00001(only incrising size of this index)
second index: my-log-00002 (having 0 doc)

Can any one help why this is happening.

2

I'm suspecting your logstash conf's output for this specific index is not pointing to ILM alias,
Could you share the logstash .conf's output?

3

Thanks for your quick response.
I am not using logstash for transferring log I am using filebeat, here is my filebeat output configuration.

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:13.xx.xx.xx}:${ELASTICSEARCH_PORT:31116}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      indices:
        - index: "%{[type]}-000001"
          when.equals:
                  type: "my-log"
    setup.ilm.enabled: false
    ilm.enabled: false
4

As I was suspected, your index is pointing to %{[type]}-000001 hence the ilm not working.
You need to point to your alias instead of an index name with the 000001.

Some useful info:

Hope this could help!

5

But my %{[type]} parameter value is ```
my-log

It means both are the same [ type] value and my alias.

so you mean to say don't need to provide index as 
index: "%{[type]}-000001" I can write like this
index: "%{[type]}".
6

If your %{[type]} match your rollover-alias it should work.

But before you do the changes, do you mind to share below info for a better understanding?

  • GET /my-log-00001 (Just wanted to see the aliases portion)
  • GET /my-log-00002 (Just wanted to see the aliases portion)
  • GET my-log-00001/_ilm/ explain
  • GET my-log-00002/_ilm/ explain
7

Now its creating 2 indexes after that its giving this error
illegal_argument_exception: index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]

GET prod-nmn-000001/_ilm/ explain

{
  "indices" : {
    "prod-nmn-000001" : {
      "index" : "prod-nmn-000001",
      "managed" : true,
      "policy" : "new_policy",
      "lifecycle_date_millis" : 1602650088714,
      "age" : "28.23m",
      "phase" : "hot",
      "phase_time_millis" : 1602650088802,
      "action" : "rollover",
      "action_time_millis" : 1602650621674,
      "step" : "ERROR",
      "step_time_millis" : 1602651221583,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]",
        "stack_trace" : """
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92)
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:274)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:175)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:203)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)

"""
      },
      "phase_execution" : {
        "policy" : "new_policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "100mb",
              "max_age" : "2h"
            },
            "set_priority" : {
              "priority" : 10
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1602499425529
      }
    }
  }
}

GET /prod-nmn-000001

{
  "prod-nmn-000001" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "flags" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "gw_timestamp" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "loglevel" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "messages" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "network-id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "new_policy",
          "rollover_alias" : "prod-nmn"
        },
        "number_of_shards" : "1",
        "provided_name" : "prod-nmn-000001",
        "creation_date" : "1602650088714",
        "priority" : "10",
        "number_of_replicas" : "0",
        "uuid" : "-E-yDN6hQFmRxpfl_gia0w",
        "version" : {
          "created" : "7040299"
        }
      }
    }
  }
}
8

It shows alias is empty for this index.

Did you bootstrap your index after you created your policy and template?

ILM require below steps:

  1. Create a lifecycle policy that defines the appropriate phases and actions. See Create a lifecycle policy above.
  2. Create an index template to apply the policy to each new index.
  3. Bootstrap an index as the initial write index.
  4. Verify indices are moving through the lifecycle phases as expected.

also, isn't the index name my-log-xxxxx, why the log shows prod-nmn-xxxxxx instead?

9

Thanks Kavier,
I changed the index name from my-log to prod-nmn. can we do it dynamically? because my index is going to create dynamically so how to do bootstrap for dynamically created index? as I check in example

PUT timeseries-000001
{
  "aliases": {
    "timeseries": {
      "is_write_index": true
    }
  }
}

they are bootstrapping only for this index timeseries-000001 what about other index which ae going to create after rollover?

10

This is where the index template come in as you configured below.

you only have to bootstap the first one, and subsequently as long as the index pattern in your index template matched, it will auto assign the rollover alias to your xxx-000002 index

I strongly suggest to have a read up on the whole ILM document as it is really helpful to understand how ILM works.
Also I find Datasteam doc helps me understand ILM even better.

Hope this can help you!

11

Thanks Kavier,

I went through the document it's help a lot to understand ILM . one small question I am giving rollover index size 100mb but my rollover index are containing more then 100 mb log like 350Mb,284Mb,456Mb. Why it's happening like this?

12

Can you run another round of below commands to have a better idea:

  • GET INDEXNAME-000001/_ilm/ explain
  • Dry run following your existing policy for that specific index
13

I applied the policy and template in a new
alias "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296"

GET 0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001/_ilm/explain

{
  "indices" : {
    "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001" : {
      "index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001",
      "managed" : true,
      "policy" : "new-device-policy",
      "lifecycle_date_millis" : 1602825221945,
      "age" : "2.38h",
      "phase" : "hot",
      "phase_time_millis" : 1602824303020,
      "action" : "complete",
      "action_time_millis" : 1602825222393,
      "step" : "complete",
      "step_time_millis" : 1602825222393,
      "phase_execution" : {
        "policy" : "new-device-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "1mb",
              "max_age" : "1h"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1602819713971
      }
    }
  }
}

I did a dry run on my alias with max age 1 day and max size 1 mb as i mention in policy.

POST /0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296/_rollover?dry_run
{
  "conditions" : {
    "max_age": "1d",
    "max_size": "1mb"
  }
}

Response

{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000009",
  "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010",
  "rolled_over" : false,
  "dry_run" : true,
  "conditions" : {
    "[max_age: 1d]" : false,
    "[max_size: 1mb]" : true
  }
}

where i am doing wrong?

14

After this one did you see the index got rollover to xxx-000002?

15

index are getting rollover by its own, now I have total 10 index after getting rollover and it keeps going. after doing a dry run in responce its printing old index and new index but logs are going to the old index, no new index is getting created immediately its happening by its own after some time.

{
  "acknowledged" : false,
  "shards_acknowledged" : false,
  "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010",
  "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000011",
  "rolled_over" : false,
  "dry_run" : true,
  "conditions" : {
    "[max_age: 1d]" : false,
    "[max_size: 1mb]" : true
  }
}
16

Dry run is a simulation, it should not execute the rollover action.

Anyway, it seems like its working already isnt it?

I dont really get this part, could you elaborate on this?

17

if it's doing a dry run then ```
acknowledged

yes, it's working but the size I have provided in policy to rollover (2mb) so  rollover index should container 2 mb log not more then that .. but my rollover index contain 25 to 60 mb logs.. why its happenig?
18

There's an interval of 10 minutes that rollover will happen

indices.lifecycle.poll_interval
(Dynamic, time unit value) How often index lifecycle management checks for indices that meet policy criteria. Defaults to 10m .

Rollover will only run after every interval, if within the interval your logs keeps coming in, it will still store at the previous even after the policy hits.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.