# ILM for each module fileset

**URL:** <https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812>\
**Category:** Beats\
**Tags:** ilm-index-lifecycle-management, filebeat\
**Created:** [August 20, 2020, 7:00pm UTC](https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812 "2020-08-20T19:00:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mweissha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mweissha/32/30469_2.png) [@mweissha](https://discuss.elastic.co/u/mweissha)\
**Post date:** [August 20, 2020, 7:00pm UTC](https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812/1 "2020-08-20T19:00:08Z")

</div>

Hey all, I'm currently in the process of porting from another logging system over to elasticsearch/kibana using filebeat as the shipper and ingest pipelines on the cloud to parse the data. What I want to accomplish is to have a seperate index for each fileset module. I'm able to do that by specifying 'index: "test-000001"' in the config file for each module fileset and I can get ILM to pick that up and rotate the log file. The trouble is that filebeat keeps pushing to the same named index with "000001" and it doesn't rotate. So what I'm looking for is help to get ILM working properly so that I can write to one index from filebeat and have ILM work to rotate the writable index. This is what I have so far for config and other settings but the ILM part is not working.

test/config/test.yml

```auto
    type: log
    paths:
    {{ range $i, $path := .paths }}
     - {{$path}}
    {{ end }}
    fields:
      tags: ["test"]
    multiline:
      pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
      negate: true
      match: after
    index: "test"

```

GET test\*/\_ilm/explain

```auto
    {
      "indices" : {
        "test" : {
          "index" : "test",
          "managed" : true,
          "policy" : "test_ilm",
          "lifecycle_date_millis" : 1597947506212,
          "age" : "10.67m",
          "phase" : "hot",
          "phase_time_millis" : 1597947506545,
          "action" : "rollover",
          "action_time_millis" : 1597947605451,
          "step" : "check-rollover-ready",
          "step_time_millis" : 1597947605451,
          "phase_execution" : {
            "policy" : "test_ilm",
            "phase_definition" : {
              "min_age" : "0ms",
              "actions" : {
                "rollover" : {
                  "max_size" : "20gb",
                  "max_age" : "1m"
                },
                "set_priority" : {
                  "priority" : 100
                }
              }
            },
            "version" : 10,
            "modified_date_in_millis" : 1597947487738
          }
        }
      }
    }

```

```auto
POST _aliases
{
  "actions" : [
    { "add" : { "index" : "test", "alias" : "test-000001", "is_write_index" : "true" } }
  ]
}

```

GET \_alias/test\*

```auto
{
  "test" : {
    "aliases" : {
      "test-000001" : {
        "is_write_index" : true
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![mweissha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mweissha/32/30469_2.png) [@mweissha](https://discuss.elastic.co/u/mweissha)\
**Post date:** [August 20, 2020, 7:07pm UTC](https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812/2 "2020-08-20T19:07:29Z")

</div>

Update:

I get an error with this current config:

```auto
illegal_argument_exception: index name [test] does not match pattern '^.*-\d+$'

```

Not sure if that is able to help or not. It makes me more confused.

---

<div class="post-metadata">

**Author:** ![mweissha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mweissha/32/30469_2.png) [@mweissha](https://discuss.elastic.co/u/mweissha)\
**Post date:** [September 4, 2020, 12:34am UTC](https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812/3 "2020-09-04T00:34:41Z")

</div>

Just in case anyone is following this or see this post i the future. I found the solution to this problem with [this thread](https://discuss.elastic.co/t/ilm-setup-dynamic-alias-in-filebeat/187298/2).

In a nutshell it describes setting up the index lifecycle settings however you want, then you add a template to your index and the key to all of this was to bootstrap the first index with URI encoding like so:

```auto
PUT /%3Cfilebeat-7.1.1-indexname-%7Bnow%2Fd%7D-1%3E {
"aliases": {
"indexname":{
"is_write_index": true
} } } 

```

The URI encoding was what was missing for my previous attempts and this is what fixed it for me. Thanks to [chandra2037](https://discuss.elastic.co/u/chandra2037) for this post.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2020, 2:35am UTC](https://discuss.elastic.co/t/ilm-for-each-module-fileset/245812/4 "2020-10-02T02:35:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.