How to fix Index Lifecycle Rollover Alias is empty or not defined

Elastic Stack Elasticsearch
1

I've been running this (uses date math) but it doesn't seem to be working:

PUT /%3Cos-linux-%7Bnow%2Fd%7D-000001%3E
{
    "aliases": {
        "os-linux": {
            "is_write_index": true
        }
    }
}

which produces: os-linux-2024.01.23-000001

I deleted all old os-linux-* indices, stopped Logstash, ran the above command, then restarted Logstash but still same "illegal_argument_exception: setting [index.lifecycle.rollover_alias] for index [os-linux-2024.01.23] is empty or not defined"

Any advice?

2

Hi @roman-tasi

Share your output section of logstash

Also, what version are you on both elasticsearch and logstash?

3

@stephenb
Logstash output block:

output {
        if [type]=="fortune" {
            elasticsearch {
            hosts => ["localhost:9200"]
            index => "os-linux-%{+yyyy.MM.dd}"
            }
        stdout { codec => rubydebug }
        }
        if [type]=="syslog" {
                if [m] in [ "1080", "745", "263", "1079" ] {
                elasticsearch {
                hosts => ["localhost:9200"]
                index => "account-access-%{+yyyy.MM}"
                }
                } else {
                elasticsearch {
                hosts => ["localhost:9200"]
                index => "syslog-%{+yyyy.MM.dd}"
                }
                }
        stdout { codec => rubydebug }
        }

        if [type]=="beats" {
                if [agent][type]=="auditbeat" {
                    elasticsearch {
                    hosts => ["localhost:9200"]
                    index => "auditbeat-%{+yyyy.MM.dd}"
                    }
                }
                if [agent][type]=="metricbeat" {
                    elasticsearch {
                    hosts => ["localhost:9200"]
                    index => "metricbeat-%{+yyyy.MM.dd}"
                    }
                }
                if [host][os][type]=="linux" {
                    #if [log][file][path]=="/var/log/audit/audit.log" {
                    #    path => "/etc/elasticsearch/audit-output.txt"
                    #}
                    if [host][name]=="mail.uhtasi.org" {
                        if [cmd]=="Auth"{
                            elasticsearch {
                            hosts => ["localhost:9200"]
                            index => "account-access-%{+yyyy.MM}"
                            }
                            } else {
                            elasticsearch {
                            hosts => ["localhost:9200"]
                            index => "zimbra-%{+yyyy.MM.dd}"
                            }
                        }
                    } else {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "os-linux-%{+yyyy.MM.dd}"
                        }
                    }
                }
                if [host][os][type]=="windows" or [agent][name]=="DCON2" or [agent][name]=="Dcon3" {
                    if [event][code] in [ "307", "4624", "4625", "4634", "4723", "4740", "4767", "11707" ] {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "account-access-%{+yyyy.MM}"
                        }
                    } else {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "winlogbeat-%{+yyyy.MM.dd}"
                        }
                    }
                }
        stdout { codec => rubydebug }
        }
}

Both ES and Logstash are version 7.17.9

4

Do you have an index template that defines which ILM policy to apply?

If so, you are not writing to the write alias which you just show above

So your output index should be

index => "os-linux"

That is how it works you write to the write alias... Not the actual concrete index.

That is how rollover works

5

What is your index template look like??

Since it does not look like you're using data streams, you should be following these instructions

6

These are the settings of my os-linux index template:

{
  "index": {
    "lifecycle": {
      "name": "os-linux-policy"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
  }
}

What I am confused about is that the other indices in my output block are following the index-%{+yyyy.MM.dd}" format without issue.

9

Hopefully I didn't accidentally close this topic.

I Can't tell you why the others because you haven't showed me all the information.

You haven't showed me enough of the template to show what indices it matches.

But I can tell you without a doubt. Absolutely certainty that when you create an alias for ILM purposes you need to write to the alias, not the concrete index if you want ILM to work

Did you carefully read the docs I linked

Do these match the index pattern in the template.

In short, those indices that you put the date on the end are daily indices. They are not managed by ILM

10

For example these are some of the settings on my zimbra-2024.01.19 :

"settings": {
    "index": {
      "lifecycle": {
        "name": "zimbra-policy"
      },
      "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_cold,data_warm,data_hot"
          }
        }
      },

It is a daily index, however it seems to be automatically assigned to zimbra-policy. Any idea why?

Edit - Screenshot showing ILM :

zimbra
zimbra907×623 36 KB

Also want to mention it doesn't get the "Index Lifecycle Rollover Alias is empty or not defined" error.

11

Can you please show the entire index template except the mappings you are leaving out key parts.

Also.. perhaps we should step back and try to understand what you were trying to accomplish.

You are mixing concepts of daily indices with ILM and ILM with rollover. It's pretty common confusion.

What are you trying to do?

And I suggest we fix one thing at a time...

I don't have enough information parts of the information...

I would need to see the full
Template
Policy
And run some other commands to determine what is going on.

So what do you want to work on first? os- or Zimbra?

If you want to fix os- I will need to see
The entire ILM policy
The entire template except the mappings

12

@stephenb - I want to fix the os-linux first.

Here's the ILM policy:

PUT _ilm/policy/os-linux-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_age": "30d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "cold": {
        "min_age": "20h",
        "actions": {
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Here's the template w/o the mappings:

{
  "index": {
    "lifecycle": {
      "name": "os-linux-policy"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
  }
}

Thanks!

13

If that is in fact the complete index template you are missing a key part

"index_patterns": ["te*", "bar*"],

It looks like you are using legacy templates here

index_patterns
(Required, array of strings) Array of wildcard expressions used to match the names of indices during creation.

Without that the created index will not match the template and thus the ILM or anything else will not be applied. I an not sure how you even created a template without it

14

@stephenb - I am running this in Dev Tools (and yes it is a legacy template):

PUT _template/os-linux
{
  "index_patterns": ["te*", "bar*"],
  "index": {
    "lifecycle": {
      "name": "os-linux-policy"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
  }
}

But it is returning this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "unknown key [index] in the template "
      }
    ],
    "type" : "parse_exception",
    "reason" : "unknown key [index] in the template "
  },
  "status" : 400
}

Any advice?

In create legacy template I just went through the required fields and got the template created (it didn't seem to ask for anything related to an array of wildcard expressions).

15

Bad syntax look at the examples further down the page..

Where you have index. Should be settings all the index settings go under that

"settings" : {
    "number_of_shards" : 1
  },

settings
(Optional, index setting object) Configuration options for the index. See Index Settings.

In Dev Tools the helper should help with syntax

You can always use the new template as well

16

I'm trying this:

PUT _template/os-linux
{
  "index_patterns": ["te*", "bar*"],
  "settings": {
    "number_of_shards": 1
  },
  "lifecycle": {
      "name": "os-linux-policy"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
}

but it is returning this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "unknown key [lifecycle] in the template "
      }
    ],
    "type" : "parse_exception",
    "reason" : "unknown key [lifecycle] in the template "
  },
  "status" : 400
}

Can you provide a command that works?

17

See if you can follow run these 1 at a time

PUT _template/os-linux
{
  "index_patterns": [
    "os-linux-*"
  ],
  "settings": {
    "lifecycle": {
      "name": "os-linux-policy",
      "rollover_alias": "os-linux"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
  }
}
  
PUT /%3Cos-linux-%7Bnow%2Fd%7D-000001%3E
{
    "aliases": {
        "os-linux": {
            "is_write_index": true
        }
    }
}  

POST os-linux/_doc
{
  "foo" : "bar"
}

# This will be whatever was created by the first command 
GET os-linux-2024.01.25-000001

POST os-linux/_rollover

POST os-linux/_doc
{
  "foo" : "bar"
}

GET os-linux-*/_search
1 Like
18

@stephenb - After running those commands, I am now getting this error for the newly created os-linux-2024.01.25 index:

illegal_argument_exception: index.lifecycle.rollover_alias [os-linux] does not point to index [os-linux-2024.01.25]

So at least it is a different error than the one posed in the original post of this thread.
Any advice on how to fix it?

19

Where and how are you getting that error?

Also those commands I provided were for you to learn the concepts / syntax no meant as ever commands.

os-linux-2024.01.25-000001

Or
os-linux-2024.01.25

20

That particular error was coming from os-linux-2024.01.25 seen in Index Management. Currently it is gone but not sure if it will return when the policy changes phases or something like that.