ILM Create indices with a date pattern in their name

Elastic Stack Elasticsearch
1

I am attempting to migrate a logstash cluster that has been running since dinosaurs roamed the earth to use ILM.

We use curator to manage our hot/warm indices and only delete data manually with human approval. I only really want ILM because some of our indices are getting quite large and it'd be better to rollover than to increase the shardcount.

I'd like to continue to have a date pattern in my indices. Both because the cluster is queried by multiple teams and I'd like to change it as little as possible and for ease of managing indices. Sometimes I need to restore old data from months ago and I am not even sure how I'd know which particular indices I need to restore. I guess you have to manually review the contents of snapshots and see which indices are new on a particular day and then guestimate how many indices you need? Sounds like a huge pain.

I can make indices rollover just fine but I can't get them to be created with a date pattern in the name.

Is this possible? Should I use some different solution?

Assorted information i feel like people will want below

heres my policy

PUT _ilm/policy/logstash-logs-windows
{
  "policy": {                       
    "phases": {
      "hot": {                      
        "actions": {
          "rollover": {             
            "max_size": "150GB",
            "max_age": "24h"
          }
        }
      }
    }
  }}

heres my template (with mappings removed for brevity)

PUT _template/logs-windows
{
  "version": 600666,
  "index_patterns": [
    "logs-windows-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "logstash-logs-windows",
        "rollover_alias": "logs-windows-ingest"
      },
      "number_of_shards": "3",
      "refresh_interval": "60s"
    }
  },
  "aliases": {
    "search_windows_logs": {}
  }
}

heres how i made my first index

PUT logs-windows-2021.08.11-000001
{
  "aliases": {
    "logs-windows-ingest": {"is_write_index": true}
  }
}

ES Version stuff

  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
    "build_date" : "2019-10-28T20:40:44.881551Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },

EDIT: what i get is a bunch of indices that all have the date on my first index i'd like to have them have the day the index was created. I'm really not picky about the index getting created at midnight I'd be open to having to restore the day before/after the day timeframe I need in situations where I need to restore old data.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.