Multiple Filebeat.yml for achieving different tasks

Suppose I have two different task to fulfill using filebeat. Is there any issues to make two different filebeat.yml for two different tasks? Its much cleaner and keeps the code separate.

Please do not ping people one minute after creating a topic.

Your question is quite generic. What exactly do you mean by 'different task'?

Regarding 'cleaner' config + different tasks, have you had a look at filebeat support for (re-)loading external configs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html#filebeat-configuration-reloading

1 Like

Thanks for the reply.

For eg:
I want to perform two different task using filebeat

First task: ship logs from /dir/abc/* to Es
Second task : get data from /dir2/a.csv to es

Now both task are different so I want them to be stored in different indices. How could that be achieved?

If you don't have this many sources, one config file should be enough. You can use one filebeat only as long as you want to send all events to the same endpoint (e.g. Logstash, Elasticsearch).

There are different strategies on configuring different indices for the different sources. One common one is to define a custom field with each input like this:

filebeat.inputs:
- type: log
  paths:
    - /dir/abc/*
  fields.task: first
- type: log
  paths:
    - /dir2/a.csv
  pipeline: process_csv
  fields.task: second

output.elasticsearch:
  hosts: [...]
  index: '%{[fields.task]}-%{[beat.version]}-%{+yyyy.MM.dd}'

This configuration will send events from the first input to the index first-6.4.0-2018-08-29 for example. Events from the second input are send to second-6.4.0-2018-08-29, but only after processing the events via the ingest node pipeline process_csv (you will have to create and maintain pipelines in Elasticsearch yourself).

If you end up with a many input configurations and the yaml file grows you can make use of external configuration support.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.