Filebeat multiple index rollover problem

Elastic Stack Elasticsearch
1

Dear all
I'm sending log data to elastic using filebeat. I have multiple files to read, actually I'm sending all log files (eg. *.log) and it works correctly, I have index and rollover works.
Now I need to have different index for different log type and I'm following
this guide

I setup the configuration:

setup.template.settings:
  index.number_of_shards: 1
setup.ilm.rollover_alias: "filebeat-test-7.10.2"
setup.ilm.enabled: true
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-test-7.10.2"
setup.ilm.overwrite: false
setup.ilm.check_exists: true



output.elasticsearch:
  hosts: ["els-mon1.local.com","els-mon2.local.com","els-mon3.local.com"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  protocol: "https"
  username: "filebeat_setup"
  password: ""
  indices:
    - index: "filebeat-test-%{[agent.version]}-api-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapi"
      setup.ilm.rollover_alias: "filebeat-test-7.10.2-api"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-test-7.10.2-api"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-test-7.10.2-api"
        pattern: "filebeat-test-7.10.2-api-*"
        enabled: true
        overwrite: false

  - index: "filebeat-test-%{[agent.version]}-noapi-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapps"
      setup.ilm.rollover_alias: "filebeat-test-7.10.2"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-test-7.10.2"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-test-7.10.2"
        pattern: "filebeat-test-7.10.2-*"
        enabled: true
        overwrite: false

I have correctly 3 different indexes
filebeat-test-7.10.2-2021.02.19
filebeat-test-7.10.2-api-2021.02.19
filebeat-test-7.10.2-noapi-2021.02.19

but only the firstone correctly apply rollover, the problem is rollover alias, It is possible to assign multiple rollover alias to the same ILM?
Or for each index I have to setup a new ilm with new Alias?

Some one have an idea to solve it?

best regards and thanks for your time

P.S. I know that many information are hardcoded into the configuration file I'll fix them when I'll solve the main problem

2

You need to keep them separate.

3

thanks @warkolm
but exactly how I can do it?

thanks for your time

4

You will need separate policies.

5

@warkolm Thanks for your answare

I changed the configuration

setup.template.settings:
  index.number_of_shards: 1
setup.ilm.rollover_alias: "filebeat-7.10.2"
setup.ilm.enabled: true
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat-7.10.2"
setup.ilm.overwrite: false
setup.ilm.check_exists: true



output.elasticsearch:
  hosts: ["els-mon1.local.com","els-mon2.local.com","els-mon3.local.com"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  protocol: "https"
  username: "filebeat_setup"
  password: ""
  indices:
    - index: "filebeat-%{[agent.version]}-api-%{+yyyy.MM.dd}"
      when.contains:
        message: "webapi"
      setup.ilm.rollover_alias: "filebeat-7.10.2-api"
      setup.ilm.enabled: true
      setup.ilm.pattern: "{now/d}-000001"
      setup.ilm.policy_name: "filebeat-7.10.2-api"
      setup.ilm.overwrite: false
      setup.ilm.check_exists: true
      setup.template:
        name: "filebeat-7.10.2-api"
        pattern: "filebeat-7.10.2-api-*"
        enabled: true
        overwrite: false

I'm trying to do it and I want that all logs with webapi into the message are stored into the filebeat-%{[agent.version]}-api-%{+yyyy.MM.dd} the other into filebeat-%{[agent.version]}-%{+yyyy.MM.dd}. I set up two different policy filebeat-7.10.2-api and filebeat-7.10.2

but I have the same problem with "api" index it does not make rollover.

Any idea?

Thanks so much

6

I implemented a solution disabling IML and creating by hand the index with correct alias

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.