# Filebeat multiple index rollover problem

**URL:** <https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [February 19, 2021, 10:27am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833 "2021-02-19T10:27:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![graimato](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@graimato](https://discuss.elastic.co/u/graimato)\
**Post date:** [February 19, 2021, 10:27am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/1 "2021-02-19T10:27:34Z")

</div>

Dear all  
I'm sending log data to elastic using filebeat. I have multiple files to read, actually I'm sending all log files (eg. \*.log) and it works correctly, I have index and rollover works.  
Now I need to have different index for different log type and I'm following  
this guide

> **[Configure the Elasticsearch output | Filebeat Reference \[7.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html)**

I setup the configuration:

```auto
setup.template.settings:
  index.number_of_shards: 1
setup.ilm.rollover_alias: "filebeat-test-7.10.2"
setup.ilm.enabled: true
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-test-7.10.2"
setup.ilm.overwrite: false
setup.ilm.check_exists: true

output.elasticsearch:
  hosts: ["els-mon1.local.com","els-mon2.local.com","els-mon3.local.com"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  protocol: "https"
  username: "filebeat_setup"
  password: ""
  indices:
    - index: "filebeat-test-%{[agent.version]}-api-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapi"
      setup.ilm.rollover_alias: "filebeat-test-7.10.2-api"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-test-7.10.2-api"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-test-7.10.2-api"
        pattern: "filebeat-test-7.10.2-api-*"
        enabled: true
        overwrite: false

  - index: "filebeat-test-%{[agent.version]}-noapi-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapps"
      setup.ilm.rollover_alias: "filebeat-test-7.10.2"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-test-7.10.2"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-test-7.10.2"
        pattern: "filebeat-test-7.10.2-*"
        enabled: true
        overwrite: false

```

I have correctly 3 different indexes  
filebeat-test-7.10.2-2021.02.19  
filebeat-test-7.10.2-api-2021.02.19  
filebeat-test-7.10.2-noapi-2021.02.19

but only the firstone correctly apply rollover, the problem is rollover alias, It is possible to assign multiple rollover alias to the same ILM?  
Or for each index I have to setup a new ilm with new Alias?

Some one have an idea to solve it?

best regards and thanks for your time

P.S. I know that many information are hardcoded into the configuration file I'll fix them when I'll solve the main problem

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 22, 2021, 12:55am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/2 "2021-02-22T00:55:52Z")

</div>

> [@graimato](#):
>
> Or for each index I have to setup a new ilm with new Alias?

You need to keep them separate.

---

<div class="post-metadata">

**Author:** ![graimato](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@graimato](https://discuss.elastic.co/u/graimato)\
**Post date:** [February 22, 2021, 11:04am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/3 "2021-02-22T11:04:55Z")

</div>

thanks @warkolm  
but exactly how I can do it?

thanks for your time

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 22, 2021, 10:28pm UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/4 "2021-02-22T22:28:17Z")

</div>

You will need separate policies.

---

<div class="post-metadata">

**Author:** ![graimato](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@graimato](https://discuss.elastic.co/u/graimato)\
**Post date:** [February 23, 2021, 1:03pm UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/5 "2021-02-23T13:03:45Z")

</div>

@warkolm Thanks for your answare

I changed the configuration

```auto
setup.template.settings:
  index.number_of_shards: 1
setup.ilm.rollover_alias: "filebeat-7.10.2"
setup.ilm.enabled: true
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-7.10.2"
setup.ilm.overwrite: false
setup.ilm.check_exists: true

output.elasticsearch:
  hosts: ["els-mon1.local.com","els-mon2.local.com","els-mon3.local.com"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  protocol: "https"
  username: "filebeat_setup"
  password: ""
  indices:
    - index: "filebeat-%{[agent.version]}-api-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapi"
      setup.ilm.rollover_alias: "filebeat-7.10.2-api"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-7.10.2-api"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-7.10.2-api"
        pattern: "filebeat-7.10.2-api-*"
        enabled: true
        overwrite: false

```

I'm trying to do it and I want that all logs with webapi into the message are stored into the filebeat-%{[agent.version]}-api-%{+yyyy.MM.dd} the other into filebeat-%{[agent.version]}-%{+yyyy.MM.dd}. I set up two different policy filebeat-7.10.2-api and filebeat-7.10.2

but I have the same problem with "api" index it does not make rollover.

Any idea?

Thanks so much

---

<div class="post-metadata">

**Author:** ![graimato](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@graimato](https://discuss.elastic.co/u/graimato)\
**Post date:** [February 26, 2021, 11:32am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/6 "2021-02-26T11:32:25Z")

</div>

I implemented a solution disabling IML and creating by hand the index with correct alias

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 26, 2021, 11:33am UTC](https://discuss.elastic.co/t/filebeat-multiple-index-rollover-problem/264833/7 "2021-03-26T11:33:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.