Output to multiple indexes straight from filebeat.yml?

Elastic Stack Beats
2

you hardcoded the index name in your output to index1. It is the index setting which selects the index name to use.

See index docs and indices docs.

The index setting supports Format Strings. That is, you can use any field in the event to construct the index.

e.g.

filebeat.prospectors:
- ...
  fields:
    type: "logs1"
- ...
  fields:
    type: "logs2"


setup.template.name: "index-%{[beat.version]}"
setup.template.pattern: "index-%{[beat.version]}-*"


output.elasticsearch:
  ...
  index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

Choosing a good index name requires some considerations. In my example I opted to have a common prefix + include the beat version and event acquisition date.
Having a common prefix ensures the index template is used for both indices. Including the beat version (or some kind of versioning) ensure your setup will not break if you ever update filebeat with changed document type (e.g. 5.4 and 6.2 template mappings are not fully copmatible). The date at the end ensure you will have a daily index. Having daily indices enables you to do some index lifecycle management. E.g. move old indices to cold storage or delete very old indices (retention policies). The index setting will read the fields.type when constructing the index name. If fields.type is missing, the default value of other will be used to construct the index name.

5 Likes