Delete indices older than 30 days using Kibana Index Lifecycle policies

[zaeemmasood]

Hello. I am new to ELK. I have managed to install and setup ELK 7.6.2 stack on RHEL 7 servers. Now as part of house keeping I need to remove/ delete indices older than 30 days to maintain certain level of available disk space.

Can I do this using Kibana console by navigating through the following:

Management => Index Lifecycle policies => Create an index lifecycle policy => Delete phase => Activate delete phase => 30 days from rollover

Are there any disadvantages of using the above?

Please note that I want to avoid using Curator (Read about it in this forum)

Thanks

[d.silwon (Daniel)]

Hi @zaeemmasood,

You can do this by the following steps:

1. Create proper policy:

PUT _ilm/policy/cleanup-history
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {}
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

2. Assign new policy to existing indexes:

PUT /index_name-*/_settings?pretty
{
  "lifecycle.name": "cleanup-history"
}

3. Create template for new indexes:

PUT /_template/logging_policy_template?pretty
{
"index_patterns": ["index_name-*", "other_index-*"], "settings": { "index.lifecycle.name": "cleanup-history" }
}

Best Regards,
Dan

[zaeemmasood]

Hello Dan. Thanks for a detailed reply. I have a few questions as follows:

1. I have 3 master nodes and 5 data nodes configured. There are three steps provided by you above for creating policy, assigning policy and creating template. Can I paste these one by one in the "Dev Tools" in Kibana and execute? Would it apply to the entire cluster?

2. Names of all my index patterns start with "prod_xxxx". When following your proposed Step 2 shall I replace:

                      PUT /index_name-*/_settings?pretty
   
                      with 
   
                      PUT /prod_*/_settings?pretty
   

3. Likewise in following your Step 3 above can I replace as follows:

"index_patterns": ["index_name-", "other_index-"], "settings": { "index.lifecycle.name": "cleanup-history" }

with

"index_patterns": ["prod_*"], "settings": { "index.lifecycle.name": "cleanup-history" }

Thanks again!

[d.silwon (Daniel)]

Hello @zaeemmasood,

Ad.1 In my opinion you can connect to one node only and execute these three steps there. In addition. In my environment I have 3 master nodes configured. My Kibana connect to all nodes because Kibana setting looks like:

server.port: 5601
server.host: "192.168.1.100"
server.name: "kibana-srv"
elasticsearch.hosts: ["http://node-01:9200", "http://node-02:9200", "http://node-03:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "Passw$Ord"

If you have any objections please check it on your one node:

curl -XGET hostname:9200/_cat/nodes

This should show you information about all cluster nodes, not for one only.

Ad.2 Correct thinking.

Ad.3 Correct thinking.

Best Regards,
Daniel

[zaeemmasood]

Thanks Dan. For your Step 3 what are the uses of template?

[d.silwon (Daniel)]

If I good remember the template is used in case of new indexes, ie. when a new index is created in Elasticsearch, then your deletion policy will be assigned to it.

In step 2 your new deletion policy will be assigned to existing indexes in Elasticsearch only.

PS. Sorry for my "broken" English.

Regards,
Dan

[zaeemmasood]

Thank you so much! Your input was very useful.

Regards,
Zaeem