Add alias to existing indices (and newly created indices) using Kibana UI

Elastic Stack Elasticsearch
1

I've set up an EFK stack on my Kubernetes cluster. I want to automatically delete indices after certain days later (i.e., log retention and rotation), so I've created an index lifecycle policy.

The policy's name is delete-after-60-days, and after hot phase, it moves to delete phase when data is 60 days old.

I've also created an index template named delete-logstash.

The index pattern is logstash-* and below are rest of the settings:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete-after-60-days",
          "rollover_alias": "logstash"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {},
    "aliases": {}
  }
}

My indices look like logstash-2023.06.01, logstash-2023.06.02, and so on, and I want to apply my index lifecycle policy to all the existing indices.

However, when I add lifecycle policy manually via Kibana UI, following error occurs:

illegal_argument_exception: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-2023.06.01]

I think I should add alias to existing indices (and of course, automatically add alias to newly created indices), but I don't know how to do this in Kibana UI.

Also, I want to ensure that any newly created indices with matching patterns (logstash-*) are correctly linked with and applied by the lifecycle policy.

Where am I doing wrong and where in the template or policy should I edit?

I would like to use Kibana UI to do this.

Thanks in advance.

2

Hi, the error you're seeing is due to the fact that the rollover alias specified in your index lifecycle policy does not point to the index you're trying to apply the policy to. In order to use index lifecycle management (ILM) with rollover, the index you're applying the policy to must be created with an alias specified in the "rollover_alias" setting.

Here's how you can fix this:

  1. First, you need to create an alias for your existing indices. You can do this by running the following command in Kibana Dev Tools:
POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "logstash-2023.06.01", "alias" : "logstash" } }
  ]
}

Replace "logstash-2023.06.01" with the name of your index. You need to do this for each index you want to apply the policy to.

  1. Next, you need to update your index template to automatically add the alias to newly created indices. You can do this by adding an "aliases" section to your template like this:
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete-after-60-days",
          "rollover_alias": "logstash"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {},
    "aliases": {
      "logstash": {}
    }
  }
}

This will ensure that any newly created indices with matching patterns (logstash-*) are correctly linked with and applied by the lifecycle policy.

  1. Finally, you can apply your lifecycle policy to your existing indices by running the following command in Kibana Dev Tools:
PUT logstash-2023.06.01/_settings
{
  "index": {
    "lifecycle": {
      "name": "delete-after-60-days"
    }
  }
}

Replace "logstash-2023.06.01" with the name of your index. You need to do this for each index you want to apply the policy to.

Remember to replace "logstash-2023.06.01" with your actual index name in the above commands. Also, please note that you can only add an alias to an index if the alias does not already exist. If the alias already exists, you will need to remove it first before you can add it to another index.

opsgpt.io assisted me with part of this answer :slight_smile:

3

Thanks a lot for your informative reply.

I've followed your advice and 1) created alias logstash to existing indices, 2) added aliases section in index template for newly created indices, and 3) applied life cycle policy to existing indices.

However, now I bump into a new error:

X Index lifecycle error

illegal_argument_exception: rollover target [logstash] does not point to a write index

I've searched for it and figured that I can set is_write_index property when adding alias. (Aliases API | Elasticsearch Guide [8.9] | Elastic)

Is it necessary to set this property as true, even when I just want to apply an index lifecycle policy to indices?

4

When I set is_write_index as true, another error occurs:

illegal_argument_exception: index name [logstash-2023.07.29] does not match pattern '^.*-\d+$'

It seems that the index name must end with number, not date, e.g., logstash-000001. Is it impossible to maintain the index name format that includes date? I want to use the index for search in Kibana UI.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.