Dynamic Beat Index with ILM

Elastic Stack Beats
, ,
1

Hello, I could use some help with dynamic indexes. I would like to create separate Filebeat indexes for a specific module, e.g. cisco.

Steps I took:

  1. Added a "when.equals" condition to my elastic output in filebeats.yml.
  2. Restarted filebeat (also ran a filebeat setup).
  3. Realised I did things out of order: missed creating the index template and bootstrapping (which I'm still foggy on as my subsequent attempts didn't work either).
   indices:
        - index: "filebeat-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"
          when.equals:
              event.module: "cisco"

This seemed to work, it created my new index just fine, but it had two issues:

  1. The ILM complained about aliases.
  2. It didn't append an increment to the index; *-000001, for example.

I've since created an index pattern to match "filebeat-7.14.0-cisco", with the following alias setting:

"index": {
    "lifecycle": {
      "name": "filebeat",
      "rollover_alias": "filebeat-7.14.0-cisco"

This is not working, as those of you who have this stuff figured out have probably guessed. I'll spare more detail, for now, to avoid providing useless information. Please let me know if anything is missing that would be helpful.

Can anyone provide guidance on the proper steps that need to be taken to accomplish having a module in its own sub-index, and still take advantage of ILM and rollover dynamically?

2

If u want the multiple dynamic indices, you'll need to create the ILM policies and index templates manually and then set the index in Filebeat to the index aliases.

1 Like
3

At this stage ILM will not work with that type of setup, you will need to follow @legoguy1000's advice sorry to say.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.