Currently using fluentbit to read in docker log files and forward on to elasticsearch and thinking about moving over to Filebeats to try to take advantage of the new ILM features.
We currently have some application specific indices being created daily and are having to manually add them into the ILM policy we have defined. We need to do this as fluentbit currently doesn't support index alias for ILM.
I've been going through the official docs today and although I can create custom indices by doing something like:-
output.elasticsearch:
hosts: ["http://localhost:9200"]
indices:
- index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "WARN"
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "ERR"
and having a template to match, it seems that ILM can only be set for a single index, e.g.
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-%{agent.version}"
setup.ilm.pattern: "{now/d}-000001"
Am I missing something here? Is it possible to remove the date portion from the indexes and also pass multiple values into setup.ilm.rollover_alias?
My actual use case doesn't use dynamic index names, or needn't have to, e.g.
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "app1-%{agent.version}", "app2-%{agent.version}"
setup.ilm.pattern: "{now/d}-000001"
output.elasticsearch:
hosts: ["http://localhost:9200"]
indices:
- index: "app1-%{agent.version}"
when.contains:
message: "app1"
- index: "app2-%{agent.version}"
when.contains:
message: "app2"
EDIT: Change naming to reflect more closely actual usage.