How to create managed indexes with current date in names?

Elastic Stack Elasticsearch
1

How to setup ILM policies with dates in the names of the indexes?

Here's the setup that works (without dates):

PUT _index_template/index-test
{
  "index_patterns": ["index-test-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "test-policy",      
      "index.lifecycle.rollover_alias": "index-test"    
    }
  }
}

PUT index-test-000001
{
  "aliases": {
    "index-test": {
      "is_write_index": true
    }
  }
}

And the filebeat.yml config which is collecting logs from the json file:

output.elasticsearch:
  indices:
    - index: "index-test"

The effect of the configuration above is that the Filebeat is sending the data to the index-test-000001 index and if the condition is met (for ex. rollover if bigger than 100MB) it creates index-test-000002, index-test-000002 and so on.

But, I want it to create indexes with the current date like this:

  • index-test-08.03.2023-000002
  • index-test-08.03.2023-000001
  • index-test-07.03.2023-000003
  • index-test-07.03.2023-000002
  • index-test-07.03.2023-000001
    etc.

I tried by adding %{+dd.MM.yyyy} to rollover_alias and Filebeat config, but it doesn't work.
Filebeat is just creating index index-test-07.03.2023 and ignores alias.

2

Hi @maar

What version of the stack and filebeat?

And do you really want indices or a data stream, have you looked into data streams?

1 Like
3

Okay, that was pretty obvious. I never used Data Streams so I caught up and this is great.
But I have a few more questions:

  1. Is it possible to reset the "counter" everyday?
    Instead of:

    .ds-index-test-2023.03.10-000010
.ds-index-test-2023.03.10-000009
.ds-index-test-2023.03.09-000008
.ds-index-test-2023.03.09-000007

    I want it:

    .ds-index-test-2023.03.10-000002
.ds-index-test-2023.03.10-000001
.ds-index-test-2023.03.09-000002
.ds-index-test-2023.03.09-000001

  2. If I will set to rollover indices after reaching 1MB in hot phase it's not precise. I have 3MB, 2MB sometimes 5MB indices, is it possible to force it to rollover exactly on 1MB?

  3. How to delete .ds* indices? I'm getting permission denied even with elastic user.

  4. How to make Data Streams "protected"? I noticed that if someone would delete Data Stream by accident all indices will be lost.

4

No, that is not possible.

Rollover just checks temporarily and works well with realistic and/ recommended index sizes. If you are testing with unrealistic and small volumes or short durations you will not get accurate results unless you change the rollover checks to happen more frequently (not recommended for production).

1 Like
5

As Christian already explained, the ILM runs on a schedule, per default it checks every 10 minutes to see if needs to take some action and this action may not happen for another 10 minutes, besides that ILM works better with sizes on tens of GB and even use sizes like this it will never be exactly the size specified, it always can be a little more depending on the event rate and event size you get.

What error are you getting? It is the only .ds* for the data stream? If so, you can't delete it because it is the current writing index of the data stream, you will be able to delete it after a rollover, but you should add a delete phase on the ILM policy instead of manually delete the .ds* indices.

The same you would protect a normal index, setting specific permissions for your users, if a user has permissions to delete an index it could accidently delete a timebased indice as well.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.