Delete indices older than 30 days using Kibana Index Lifecycle policies

Hello. I am new to ELK. I have managed to install and setup ELK 7.6.2 stack on RHEL 7 servers. Now as part of house keeping I need to remove/ delete indices older than 30 days to maintain certain level of available disk space.

Can I do this using Kibana console by navigating through the following:

Management => Index Lifecycle policies => Create an index lifecycle policy => Delete phase => Activate delete phase => 30 days from rollover

Are there any disadvantages of using the above?

Please note that I want to avoid using Curator (Read about it in this forum)

Thanks

Hi @zaeemmasood,

You can do this by the following steps:

  1. Create proper policy:
PUT _ilm/policy/cleanup-history
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {}
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
  1. Assign new policy to existing indexes:
PUT /index_name-*/_settings?pretty
{
  "lifecycle.name": "cleanup-history"
}
  1. Create template for new indexes:
PUT /_template/logging_policy_template?pretty
{
"index_patterns": ["index_name-*", "other_index-*"], "settings": { "index.lifecycle.name": "cleanup-history" }
}

Best Regards,
Dan

1 Like

Hello Dan. Thanks for a detailed reply. I have a few questions as follows:

  1. I have 3 master nodes and 5 data nodes configured. There are three steps provided by you above for creating policy, assigning policy and creating template. Can I paste these one by one in the "Dev Tools" in Kibana and execute? Would it apply to the entire cluster?

  2. Names of all my index patterns start with "prod_xxxx". When following your proposed Step 2 shall I replace:

                       PUT /index_name-*/_settings?pretty
    
                       with 
    
                       PUT /prod_*/_settings?pretty
    
  3. Likewise in following your Step 3 above can I replace as follows:

"index_patterns": ["index_name-", "other_index-"], "settings": { "index.lifecycle.name": "cleanup-history" }

with

"index_patterns": ["prod_*"], "settings": { "index.lifecycle.name": "cleanup-history" }

Thanks again!

Hello @zaeemmasood,

Ad.1 In my opinion you can connect to one node only and execute these three steps there. In addition. In my environment I have 3 master nodes configured. My Kibana connect to all nodes because Kibana setting looks like:

server.port: 5601
server.host: "192.168.1.100"
server.name: "kibana-srv"
elasticsearch.hosts: ["http://node-01:9200", "http://node-02:9200", "http://node-03:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "Passw$Ord"

If you have any objections please check it on your one node:

curl -XGET hostname:9200/_cat/nodes

This should show you information about all cluster nodes, not for one only.

Ad.2 Correct thinking.

Ad.3 Correct thinking.

Best Regards,
Daniel

Thanks Dan. For your Step 3 what are the uses of template?

If I good remember the template is used in case of new indexes, ie. when a new index is created in Elasticsearch, then your deletion policy will be assigned to it.

In step 2 your new deletion policy will be assigned to existing indexes in Elasticsearch only.

PS. Sorry for my "broken" English.

Regards,
Dan

Thank you so much! Your input was very useful.

Regards,
Zaeem

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.