How to configure my index life cycle policy

Hi,

I am using ELK stack version 7.1.0 in ubuntu 14.04 and java 8 envinorment.

I am new to this 7.1.0 Previously i used ELK 6.8.0 version.
Previously my logstash used to create an index everyday automatcially in ELK 6.8.0 version , but now all my data is getting into only one index and i even tried changing my logstash.conf file but it isn't working

I am thinking that's because of my logstash-policy
in index life cycle policy.

Please help configure that in a way it creates new index eachday with that current day date and delete that index after 10 days from index creation.

logstash.conf

input {
file {
path => "/home/i.log"
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => ["message","%{TIMESTAMP_ISO8601:date}*%{GREEDYDATA:messagedata}"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
user => elastic
password => *******
}
stdout { codec => rubydebug }
}

Index name getting created by policy

index life cycle policy

The index deletion is also didn't happended I tried by changing it into 1 hour.

Please help me configure it.

Hey ,

Can someone please look into to this ?

Hi @Vamsi_Vutukuri

First of all to create new index each day I recommend you this output configuration:

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
manage_template => false
user => elastic
password => *******
}
}

This one works for me.

And now to delete all index older than 10 days with lifecycle I recommend you make it using the dev tools:

First create the lifecycle policy:

PUT _ilm/policy/policy_name
{
"policy": {
"phases": {
"delete": {
"min_age": "10d",
"actions": {
"delete": {}
}
}
}
}
}

This policy will delete all the index when will be older than 10 days (you can change the policy_name)

Then you have to create a template for your index with the policy that we have created (this is probably the reason why your policy didnt work):

PUT _template/my_template
{
"index_patterns": ["logstash-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "policy_name",
"index.lifecycle.rollover_alias": "test"
}
}

This template will be aplicated to all the index called "logstash-*" so will work for you (you can change the values in strong as you want)

And thats all. All the new index will be deleted after 10 days, you can check if the template and the lifecycle policy have been apllied as I show you in the screenshots:

image

If have any doubt tell me and if want more information about the different phases you can take a look this: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

@dgonzalezp Thank you soooooooooooooo much !

Your solution worked .

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.