ILM with multiple indices

Elastic Stack Elasticsearch
1

Hello,

I am working to set-up a hot-warm-cold-delete policy for my ELK cluster that has multiple indices. I basically have 4 questions and as much data following them up as I thought needed.

Many thanks to any help!

All data ingested into ES comes through Logstash first. Example LS config:

output {
 if [type] == 'syslog' {
  elasticsearch {
        hosts => [ "10.15.1.108:9200" ]
        ilm_enabled => true
        ilm_rollover_alias => "wincollect-ilm"
        index => "wincollect-%{+YYYY.MM.dd}"
        user => "****"
        password => "****"

                 }

                       }
       }

I am trying to follow the many guides and blog posts out there. Before implementing ILM I did not have either of the ilm_ options and thus an index titled wincollect-date was created. I have learned that date formatting is difficult with ILM so I created the alias. I don't need date formatting anyway.

  1. For creating the index template, do I need to create a template for each index and then specifiy the template in each different logstash config?

For the above LS config, would I create this template:

PUT _template/wincollect-template
{
  "index_patterns": ["wincollect-"], 
  "settings": {
    "index.lifecycle.name": "hot-warm-cold", 
    "index.lifecycle.rollover_alias": "wincollect" 
  }
}
  1. And then add this to the LS config?

ilm_policy => "wincollect-template"

  1. And, do I need to do anything with POST /_aliases? ie
POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "wincollect-*",
        "alias": "wincollect-ilm"
      }
    }
  ]
  
}
  1. OR, is ilm_policy supposed to be the actual policy I created? In this case, "hot-warm-cold" as seen below:
{
  "hot-warm-cold" : {
    "version" : 2,
    "modified_date" : "2019-10-14T13:40:50.345Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "60d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "365d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "cold"
              }
            },
            "freeze" : { },
            "set_priority" : {
              "priority" : 0
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "100gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "2190d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}
2

Figured it out.

Following

and

I first created my policy using Kibana. Cross checked with the devconsole using

GET /_ilm/policy

In the devconsole, I made a template for each of my indexes:

PUT _template/wincollect-template
{

"index_patterns": ["wincollect-"],
"settings": {
"index.routing.allocation.require.data": "hot",
"index.lifecycle.name": "hot-warm-cold",
"index.lifecycle.rollover_alias":"wincollect-ilm"
}
}

I made 5 such templates.

I then had to delete some preexisting indexes. I could have "renamed" them, but the loss of data did not matter at this time. Next I did the whole bootstrapping thing.

PUT wincollect-ilm-000001
{
"aliases": {
"wincollect-ilm":{
"is_write_index": true
}
}
}

I removed all aliases that i had created manually and let them be created by the template. I think there was one index that didn't fit the template that I forced an alias on.

2 weeks later, still working as expected.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.