# Elasticsearch custom ILM

**URL:** <https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [August 18, 2019, 5:35pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659 "2019-08-18T17:35:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![darrylrmiles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darrylrmiles/32/74205_2.png) [@darrylrmiles](https://discuss.elastic.co/u/darrylrmiles)\
**Post date:** [August 18, 2019, 5:35pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/1 "2019-08-18T17:35:09Z")

</div>

Hello,

I am attempting to setup custom indexes for winlogbeat via logstash using ILM settings in the elasticsearch output. I would like to use the default template for winlogbeat (winlogbeat-7.3.0) but have different custom indexes for each organization. The purpose of the separate indexes is to have users be able to login to the kibana instance with roles that can only view their particular indexes for winlogbeat. I would like to use the following base structure for the indexes

winlogbeat-{agent.version}-organization

I figured appending the organization to the end would allow me to use the default template for winlogbeat according to each version as well as to use the built in dashboards and visualizations for winlogbeat-\*. Also, it would allow me to use a default ILM policy for all organization indexes just by specifying the index pattern in a specific ILM template similar to how it is described here

> **[Implementing Hot-Warm-Cold in Elasticsearch with Index Lifecycle Management](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management)**
>
> Index lifecycle management (ILM) is part of Elasticsearch and is designed to help you manage your indexes. In this blog, we will explore how to implement a hot-warm-cold architecture using ILM. Hot-warm-cold architectures are common for time series...

My problem is mainly when specifying the output for ilm\_rollover\_alias in logstash's elasticsearch output it is not appending the date and increment number like I would hope. Here is the logstash config:

```
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "elastic_cloud_url"
    user => "user"
    password => "password"
    ilm_enabled => true
    ilm_rollover_alias => "%{[@metadata][beat]}-%{[@metadata][version]}-miles"
    ilm_policy => "hot-warm-delete-30days"
  }
}

```

So it is creating the index and applying the correct ILM policy but the index name is just what is in the rollover alias without the ilm\_pattern appended to the end:

winlogbeat-7.3.0-miles

It also seems to be using the correct winlogbeat-7.3.0 template from what I can tell on the index settings since I configured that one in particular to have 5 shards and 1 replica. Below is snippet pulled from index settings via kibana:

```
{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "hot-warm-delete-30days",
        "rollover_alias": "winlogbeat-7.3.0"
      },
      "routing": {
        "allocation": {
          "require": {
            "data": "hot"
          }
        }
      },
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "5",
      "provided_name": "winlogbeat-7.3.0-miles"

```

I noticed in the snippet above also that the rollover\_alias is the default from the winlogbeat-7.3.0 template so something doesn't seem correct. I am getting the error in kibana

illegal\_argument\_exception: index.lifecycle.rollover\_alias [winlogbeat-7.3.0] does not point to index [winlogbeat-7.3.0-miles]

Am I able to structure my data and policies in this way or am I way off base here?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [August 18, 2019, 9:21pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/2 "2019-08-18T21:21:03Z")

</div>

You can do it, it's similar to what I'm doing.... getting there is all the fun.

Did you run winlogbeat setup from winlogbeat as configured? I think you need a template for each index name pattern. I think running setup for each config would create a more-qualified index name.

Here's how I did our setup, there are any ways...

1. I created the default beats template
2. I created a higher priority template for each unique index, but it is small, just adds ILM settings to the new index.
3. In beats elasticsearch output, I used index name instead of the ilm\_ values. That let me use variables where you have "miles".
4. I created a "boot-strap" index each time, before starting beats for that index.

---

<div class="post-metadata">

**Author:** ![darrylrmiles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darrylrmiles/32/74205_2.png) [@darrylrmiles](https://discuss.elastic.co/u/darrylrmiles)\
**Post date:** [August 18, 2019, 10:18pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/3 "2019-08-18T22:18:27Z")

</div>

I think you set me on the right track. I ended up finding this article regarding apm manual index lifecycle management

[https://www.elastic.co/guide/en/apm/server/current/manual-ilm-setup.html](https://www.elastic.co/guide/en/apm/server/current/manual-ilm-setup.html)

I did load the default beats template originally so it has order of 1. I ended up adding another template specifically for my custom index

PUT \_template/winlogbeat-7.3.0-miles-ilm  
{  
"order": 2,  
"index\_patterns": ["winlogbeat-7.3.0-miles\*"],  
"settings": {  
"index.lifecycle.rollover\_alias": "winlogbeat-7.3.0-miles",  
"index.lifecycle.name": "hot-warm-delete-30days"  
}  
}

I then manually added the index with the alias association (don't believe I need date in index for this to work)

PUT winlogbeat-7.3.0-miles-000001  
{  
"aliases": {  
"winlogbeat-7.3.0-miles": {  
"is\_write\_index": true  
}  
}  
}

So now logstash elasticsearch output is still using the ilm\_rollover\_alias with metadata to form the value winlogbeat-7.3.0-miles for this beat and is writing to the newly created index.

```
  "settings": {
    "index": {
      "lifecycle": {
        "name": "hot-warm-delete-30days",
        "rollover_alias": "winlogbeat-7.3.0-miles"
      },
      "routing": {
        "allocation": {
          "require": {
            "data": "hot"
          }
        }
      },
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "5",
      "provided_name": "winlogbeat-7.3.0-miles-000001"

```

Seems like this will work but I feel like i'm missing something that should make this a bit easier and more dynamic than the manual method

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [August 18, 2019, 11:21pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/4 "2019-08-18T23:21:25Z")

</div>

I had a somewhat related case with Elastic. Paraphrased, with ILM, there is setup, it's just the cost of ILM.

It seems from other posts here that missing the manual creation of the initial index is often missed.

Having a date in the index is a tradeoff, and to some extent, adds confusion. It is the date the index is created, if it takes many days to rollover, it will of course contain other events from later dates. We are using dates since we started with the old style daily indices.

Finally, you will have to prepare the system before you let a future beats version. We are to try to only support selected clients, not every one, then encourage timely client migration to the new target version.

---

<div class="post-metadata">

**Author:** ![darrylrmiles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darrylrmiles/32/74205_2.png) [@darrylrmiles](https://discuss.elastic.co/u/darrylrmiles)\
**Post date:** [August 18, 2019, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/5 "2019-08-18T23:59:45Z")

</div>

Thanks for the explanation. I think that the time spent to do the up front setup is a good trade off to having better index management.

Thanks again for the direction and help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2019, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-custom-ilm/195659/6 "2019-09-15T23:59:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.