Output to multiple indexes straight from filebeat.yml?

[vopsec (Vincent)]

I'm trying to set up filebeat to ingest 2 different types of logs. They're in different locations and they should output to different indexes. This is what I have so far:

filebeat.prospectors:

- type: log

  enabled: true
  paths:
    - /var/log/index1/*
  json.add_error_key: true
  json.message_key: log
  #json.keys_under_root: true

  pipeline: geoip

- type: log
  paths:
    - /usr/local/index2/*.log
  fields:
    type: "index2"
  fields_under_root: true

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.enabled: false
setup.template.name: "index1"
setup.template.pattern: "index1-*"

#================================ General =====================================

name: u2-filebeat

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]

  protocol: "http"

  index: "index1"

  pipeline: geoip

#================================ Logging =====================================

#logging.level: debug

I don't think I'm doing this right... In addition, I've never configured logstash so I'm at a loss as to whether not or using logstash is necessary. I've always simply sent this data directly to Elasticsearch on port 9200.

Any help would be immensely appreciated! Thank you!

[steffens (Steffen Siering)]

you hardcoded the index name in your output to index1. It is the index setting which selects the index name to use.

See index docs and indices docs.

The index setting supports Format Strings. That is, you can use any field in the event to construct the index.

e.g.

filebeat.prospectors:
- ...
  fields:
    type: "logs1"
- ...
  fields:
    type: "logs2"


setup.template.name: "index-%{[beat.version]}"
setup.template.pattern: "index-%{[beat.version]}-*"


output.elasticsearch:
  ...
  index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

Choosing a good index name requires some considerations. In my example I opted to have a common prefix + include the beat version and event acquisition date.
Having a common prefix ensures the index template is used for both indices. Including the beat version (or some kind of versioning) ensure your setup will not break if you ever update filebeat with changed document type (e.g. 5.4 and 6.2 template mappings are not fully copmatible). The date at the end ensure you will have a daily index. Having daily indices enables you to do some index lifecycle management. E.g. move old indices to cold storage or delete very old indices (retention policies). The index setting will read the fields.type when constructing the index name. If fields.type is missing, the default value of other will be used to construct the index name.

[vopsec (Vincent)]

@steffens

index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}

Just tried this and it works!! Thank you for the assistance and the tips surrounding having a date attached to the end of index.