Are date indexes not allowed anymore?

Elastic Stack Elasticsearch
1

My indexes are created from a template as ‘name-year.month.date’. It seems that this isn’t acceptable for index-lifecycle-policies anymore. I get this error “index name [name-2025.12.09] does not match pattern '^.*-\d+$'“.

At one point this process was working. I was able to rollover/delete indexes based on age.

Is there a workaround?

2

That regular expression pattern does as far as I can see not appear to match the index name so I would recommend fixing the pattern. (You need to account for the dots in the date) If this at some point has worked that must have been a bug that has been fixed.

3

Is there something I can do to change the regex pattern? I rather like having a date stamp on my indexes.

4

There are 2 things related to date pattern indices.

The first is when indices roll over, many have their birthdate in the name. This is still done by many of the Elastic provided integrations.

The second type is where the date from the event is used in the index name. That causes a new index to be created daily. That can lead to a lot of smaller indices, which can cause problems. Another problem that can happen is a new beats/agent starts harvesting host logs that go back for years and the config doesn’t block the older events (I don’t remember the parameter). If you have 2 years of data, you can suddenly create 720 new indices. (I’ve done it, not fun)

5

I need them to rollover daily. Beats create a large amount of data and I need to be able to prune it by date on the regular. This is what I was trying to achieve with the index-lifecycle.

Otherwise it’s a bash script. Boo.

6

For the regex pattern I think something like ^.*-[\d\.]+$ might work, but I have not tested it. That pattern should look for a sequence of dots and digits to end the index name.

How many different time based indices are you creating?

What is your retention period for these?

7

How are you indexing your data? With Logstash or beats?

Daily indices are not good practice as they can lead to big or small indices, it would be better to rollover by size.

In this case if you want to keep daily indices I would not use ILM to rollover them, just to delete.

You can have a policy that only deletes the index after some time.

1 Like
8

Data goes from beat → kafka → logstash → elastic. Logstash creates the indexes.

I tried creating a simple policy to delete but got the regex error that started this thread.

Im experimenting with data streams to see if they are more helpful w/re lifecycle.

9

So you are creating the indices with something like index => indexName-%{yyyy.MM.dd} right?

You need to create a policy with rollover disabled and configure the delete phase.

Something like this:

image
image2328×1166 153 KB

PUT _ilm/policy/<policyName>
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {},
        "min_age": "0ms"
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": false
          }
        }
      }
    }
  }
}

This will delete the index after 7 days of the creation.

If you want to have the dates on the index name, it will add some more complications, first data streams require rollover by size or age, you will not have daily indices anymore, also the date of the backing index will be the date when it rolled over.

Even if you rollover every 24 hours, it is counted by age, not like logstash that creates a new index when the day changes.

Another issue is that if you use custom naming, Logstash per default does not support writing data to custom data streams, it would require your indices to start with logs-* for example, which forces you to be careful with the template because there is a built-in template that would match logs-*.

You can have custom data streams names with Logstash, I have some, but you need to use an output like this:

output {
    elasticsearch {
        user => "username"
        password => "password"
        index => "custom-datastream-name"
        action => "create"
        compression_level => 3
        data_stream => false
        manage_template => false
        ilm_enabled => false
    }
}
1 Like
10

Ok. I disabled rollover and started some new indexes. Hopefully this will fix the problem.

W/re streams - I have this setting:

"lifecycle": {
    "enabled": true,
    "data_retention": "7d"
}

looks like it’s deleting the indexes under the streams but not the streams. Is this something that can be set?

11

This is the data stream retention, a different way of managing retention on data streams.

I'm not sure how this works as I do not use it, but it seems that the retention is configured in the index settings instead of using an ILM.