Ilm question/troubleshooting

[Mark_S (Baldrick)]

My ilm policy does not work, although I have created a similar one a couple of weeks before in another cluster and it is working just fine..

Here is what I did:

1. I pointed Logstash towards ind_alias
2. Created index template that will be used for creating the indices (see bellow)
3. Bootstrap index with a PUT command (see bellow), but i get an "invalid alias name" error (already exists). It exists because Logstash sends data and so the index is created (so of course it exists! I am wondering how this worked before...)

A desparate next step would be stopping all Logstash (there are many VMs sending so If possible i would prefer to avoid this). And starting from the beginning... But I would like to understand what am i doing wrong

Here are the details
Template:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "Mypolicy",
          "rollover_alias": "ind_alias"
        },
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "field3": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "aliases": {
      "ind-search-all": {}
    }
  }
}

ilm:

{
  "Mypolicy" : {
    "version" : 1,
    "modified_date" : "2023-04-26T10:02:14.694Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "1gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "30d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "ind_alias"
      ],
      "data_streams" : [ ],
      "composable_templates" : [
        "ind_template"
      ]
    }
  }
}

Bootstrap command:

PUT ind_alias-000000
{
  "aliases": {
    "ind_alias": {
      "is_write_index": true
    }
  }
}

This bootstrap command gives the error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "invalid_alias_name_exception",
        "reason" : "Invalid alias name [ind_alias]: an index or data stream exists with the same name 

[eMitch (Eddie Mitchell)]

If the index already exists and there is data in the ind_alias index - are you concerned about the data?

If you want to keep the data, then it may be best to err on the side of caution and stop Logstash in order to reindex your data into a new index. Then remove the old index and replace it with your bootstrapped index.

If you're not concerned with the data and are ok with the data loss, then you can use the Alias API to drop the index and add the new index with corresponding alias in an atomic way:

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "ind_alias-000000",
        "alias": "ind-alias",
        "is_write_index": true
      }
    },
    {
      "remove_index": {
        "index": "ind_alias"
      }
    }
  ]
}

Another option, if you don't want to stop the Logstash instances, is to mark the current ind_alias index as read-only. Logstash will no longer be able to send to it and will continue with retries. This is entirely dependent on many factors - including your own SLAs, how Logstash is configured, how much data is being ingested, how large the index is, etc.

[Mark_S (Baldrick)]

@eMitch : Thanks for the answer! I dropped the index and added a new one &the alias. I was not familiar with the way POST _aliases {"actions" work: It must be executing first the remove and then the add, without letting any logstash input interfere during these actions (because it worked perfectly without having to stop logstash)