Change Index name with enabled ILM

[Kosodrom (Kosmodrom)]

Hi guys,

from this documentation: Configure the Elasticsearch output | Filebeat Reference [7.4] | Elastic i learned:

The index setting is ignored when index lifecycle management is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Configure index lifecycle management to learn how to change the index name.

When I now follow the link I cannot find any information related to the index name. I see how I can change template name, pattern, alias but no index name. Or do I understand the ILM behaviour too little so I cannot read the information somehow between the lines?

Can you please point me to the part where I can learn how to change index name, while keeping the automatic ILM settings that filebeat provides out of the box?

THank you

[Kosodrom (Kosmodrom)]

Do I understand right that using the Filebeat's own ILM control I am only able to change the alias? Would it mean that all logs that i push with filebeat will be indexed in one index ?

[Adriann (Adrian)]

Hi Kosodrom,

I am not sure if I get your question right. But from what understand so far an index name and index alias is a sort of the same thing. The index can have many aliases.

Filebeat.yml

setup.ilm.rollover_alias: "filebeat-netflow"
setup.ilm.pattern: "{now/d}-rolled"
setup.ilm.check_exists: true
setup.ilm.policy_file: /etc/filebeat/ilm_policy_netflow.txt

I probably have some error in my ilm policy file (that's why there is index from 05.25 and rolled one) but nether then less you can see after an index is rolled the alias/name is changed.

You can read more about index aliases here

[Kosodrom (Kosmodrom)]

Adriann, thanks for your answer.

As far as I understood index name sort of unique identifier of your index and alias is well .. alias.

Let's for example say I have 3 different log sources, that has nothing to do with each other and I want all ship all of them via filbeat (3 filebeats. One filebeat per source). That would mean to me (as far as I understand it right now) that all these logs will be written in one single index. I could disable this behavior in filebeat of course and define templates, ilm policy and index name on my own, but lets assume I want to use the standard filebeat way.

So at the end I have 1 index with 3 totally different sources inside. I could now create some aliases but still those aliases would exist just for this one index. If this is confirmed true I need to disable this if the alias configuration really behaves as a totaly indepandant index I will keep this configuraiton.

[michielM (Michiel)]

I'm not sure if I got your question right.
Do you want every source to have a different index name?
If so I used this configuration for it at the bottom of my auditbeat.yml
#=============================== Index management =============================
setup.ilm.enabled: false

output.elasticsearch.index: "auditbeat-customname-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "auditbeat-customname"
setup.template.pattern: "auditbeat-customname-"
setup.dashboards.index: "auditbeat-cutsomname-"

The problem I am facing with this currently is the fact that because I disabled ILM, my indexes are created daily, so if you have 3 sources with 3 indexes, you would have 3 new indexes every day... This is a problem I am trying to solve at this point. I hope this helps!

[Adriann (Adrian)]

To split data from any module to different indexes you can use below syntax

output.elasticsearch:
      indices:
        - index: "filebeat-netflow-%{+yyyy.MM.dd}"
          when.equals:
            event.module: "netflow"

        - index: "filebeat-cisco-%{+yyyy.MM.dd}"
          when.equals:
            event.module: "cisco"

Here you can read more about that

[Kosodrom (Kosmodrom)]

That's exactly what I want to achive: 3 new indeces everyday. One index everyday per source. Actually my assumption was to do the exact same configuration as you did, but in the documentation there is this:

The index setting is ignored when index lifecycle management is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name.

Did you disabled ILM in your auditbeat?

Same question goes to Adriann: Did you disabled ILM in order to be able to set the index name in the output?

Thanks guys

[michielM (Michiel)]

Yes, I disabled ILM in auditbeat, it's the first line of the config I uploaded above

[Adriann (Adrian)]

I did not and at some point it was working fine, now it's a mess.