Specify ILM policy for Multiple Elasticsearch outputs

[TonyLuc]

Scenario
I am using filebeat to collect logs from 2 different sources.
It is going to 2 different indexes. Thus in the filebeat.yml the following are added to the Elasticsearch output:

output.elasticsearch:
hosts: ["someserver:9200"]
indices:
- index: "iis-%{[agent.version]}-%{+yyyy.MM.dd}"
when.equals:
event.module: "iis"
index: "logs-%{[agent.version]}-%{+yyyy.MM.dd}"

I have switch off
setup.ilm.enabled: false

My intentions (using the above example) is to save any logs from IIS to indices starting with “iis-”, while the defaults go to indices starting with “logs-”.
My question is: I want to use different ILM policies for different indices (the retention period will be different), how to I associated the ILM policy name to each index?

Any advises?
Thanks!

[rugenl (Len Rugen)]

Create a template for each index with the desired ILM policy.

It looks like you are going to create daily indices, so aren't using ILM for rollover. If you were, it's a little more complicated

[TonyLuc]

I am just using ILM just for deleting indices after a certain period.

The behavior is based on version 7.4.2
I have thought about using the template but I have problem loading the initial template.
Let me explain, when you setup a filebeat agent using IIS module and specified the template and index name and having only one elasticsearch.output as follow:

setup.template.name: "baa-%{[agent.version]}"
setup.template.pattern: "baa-%{[agent.version]}-*"

hosts: ["blacksheep:9200"]
index: "baa-%{[agent.version]}-%{+yyyy.MM.dd}"

ElasticSearch will automatically create a template called “baa-7.4.2”.
Question here is how I load this manually? This is first required to allow me to add the ILM policy name.

In other observations
I have updated from version 7.3.2 to 7.4.2, the following behaved differently.
In a multiple elastic outputs scenario as follows:

setup.template.name: "iis-%{[agent.version]}"
setup.template.pattern: "iis-%{[agent.version]}-*"

setup.template.name: "logs-%{[agent.version]}"
setup.template.pattern: "logs-%{[agent.version]}-*"

output.elasticsearch:
hosts: ["someserver:9200"]
indices:

• index: "iis-%{[agent.version]}-%{+yyyy.MM.dd}"
  when.equals:
  event.module: "iis"
  index: "logs-%{[agent.version]}-%{+yyyy.MM.dd}"

In version 7.3.2 both templates iis.7.3.2 and logs.7.3.2 were created.
In version 7.4.2 only template logs.7.3.2 was created.
Did anyone notice this?

[rugenl (Len Rugen)]

Since we use Logstash, I don't use the beats setup after initial setup. (I don't what to give the access necessary to manage templates to things that are just sending logs)

My method.... I setup the default template for each version, like "filebeat-7.4.0", by default, it has Order:1. We name our indices so that they match this index pattern, with suffixes to separate as needed, for example "filebeat-7.4.0-thing1-campus1".

If I needed a template "baa-7.4.0-*", I would just GET the default template, edit it, and POST it back using devtools or curl, but I keep the ILM part out of these templates as below.

I can create a template for index patterns "filebeat*thing1" to add mappings and I use order: 100 to trump any future field name conflicts from Elastic. This template doesn't repeat anything from the default template.

I can also create a template for index patterns "*-campus1" to add ILM options for those with order:150.

The beats setup changes from 6.x to 7.x are confusing. I'm not using 7.4.2 yet, so maybe they get more confusing. That and using Lostash are reasons why I just did it myself :-).

Good luck

[TonyLuc]

Thanks for the helps, Len.

1. I am using yours suggestion by first exporting the default template from filebeat.

Reference: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html
Powershell script: .\filebeat.exe export template --es.version 7.4.2 | Out-File -Encoding UTF8 filebeat.template.json

2. Add the policy name to the JSON file.
3. Post it to Elastic as a template “baa-7.4.2”
4. Make sure setup.template.overwrite: false. I been burnt too many times, I find it less of a headache to set it explicitly than relying on the default.
5. Set my index as index: "baa-%{[agent.version]}-%{+yyyy.MM.dd}"

My life should be good till I have to remember to repeat it again for the future upgrades.