# Multiple Filebeat.yml for achieving different tasks

**URL:** <https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 29, 2018, 7:10pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573 "2018-08-29T19:10:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Divit\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/divit_sharma/32/32348_2.png) [@Divit\_Sharma](https://discuss.elastic.co/u/Divit_Sharma)\
**Post date:** [August 29, 2018, 7:10pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573/1 "2018-08-29T19:10:58Z")

</div>

Suppose I have two different task to fulfill using filebeat. Is there any issues to make two different filebeat.yml for two different tasks? Its much cleaner and keeps the code separate.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 29, 2018, 7:21pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573/3 "2018-08-29T19:21:44Z")

</div>

Please do not ping people one minute after creating a topic.

Your question is quite generic. What exactly do you mean by 'different task'?

Regarding 'cleaner' config + different tasks, have you had a look at filebeat support for (re-)loading external configs: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html#filebeat-configuration-reloading](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html#filebeat-configuration-reloading)

---

<div class="post-metadata">

**Author:** ![Divit\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/divit_sharma/32/32348_2.png) [@Divit\_Sharma](https://discuss.elastic.co/u/Divit_Sharma)\
**Post date:** [August 29, 2018, 7:31pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573/4 "2018-08-29T19:31:57Z")

</div>

Thanks for the reply.

For eg:  
I want to perform two different task using filebeat

First task: ship logs from /dir/abc/\* to Es  
Second task : get data from /dir2/a.csv to es

Now both task are different so I want them to be stored in different indices. How could that be achieved?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 29, 2018, 10:37pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573/5 "2018-08-29T22:37:36Z")

</div>

If you don't have this many sources, one config file should be enough. You can use one filebeat only as long as you want to send all events to the same endpoint (e.g. Logstash, Elasticsearch).

There are different strategies on configuring different indices for the different sources. One common one is to define a custom field with each input like this:

```auto
filebeat.inputs:
- type: log
  paths:
    - /dir/abc/*
  fields.task: first
- type: log
  paths:
    - /dir2/a.csv
  pipeline: process_csv
  fields.task: second

output.elasticsearch:
  hosts: [...]
  index: '%{[fields.task]}-%{[beat.version]}-%{+yyyy.MM.dd}'

```

This configuration will send events from the first input to the index `first-6.4.0-2018-08-29` for example. Events from the second input are send to `second-6.4.0-2018-08-29`, but only after processing the events via the ingest node pipeline `process_csv` (you will have to create and maintain pipelines in Elasticsearch yourself).

If you end up with a many input configurations and the yaml file grows you can make use of [external configuration support](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html#filebeat-configuration-reloading).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2018, 10:37pm UTC](https://discuss.elastic.co/t/multiple-filebeat-yml-for-achieving-different-tasks/146573/6 "2018-09-26T22:37:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.