# Include files inside filebeat config file **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 18, 2016, 10:28am UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518 "2016-11-18T10:28:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Aditya\_Srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_srivastava/32/43287_2.png) [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)\ **Post date:** [November 18, 2016, 10:28am UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/1 "2016-11-18T10:28:53Z")
I have 100 log files and I want to read those files, put some fields to it and send it to logstash. If I create say 100 different prospectors all in one config file, it becomes very difficult to manage. Therefore, i want to know if there is any functionality in filebeat to include multiple config file inside main config file.say something like rsyslog?? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 18, 2016, 11:37am UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/2 "2016-11-18T11:37:49Z")
filebeat supports glob-patterns to configure multiple files in one prospector. Also `paths` setting is an array taking multiple glob-patterns. You plan to add the very same fields to all files? How is your directory/naming layout? ---
**Author:** ![Aditya\_Srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_srivastava/32/43287_2.png) [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)\ **Post date:** [November 18, 2016, 11:50am UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/3 "2016-11-18T11:50:05Z")
No fields in all the log files will be different. I will write the data into kafka from FB. say for 100 logs I want to create 100 diff topics in kafka. So the FB config file will become very difficult to understand as every thing will be written under same config file. So I want to break one big config file into multiple small config files. It is similar to what happens in rsyslog. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 18, 2016, 12:42pm UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/4 "2016-11-18T12:42:38Z")
with no fields being different, you will need only one prospector. How exactly is your directory layout and how do you plan to select the kafka topic name? All beats support multiple `-c` flags to combine multiple configurations. Plus filebeat can load additional prospector configurations via [filebeat.config\_dir](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-global-options.html#_config_dir) setting. ---
**Author:** ![Aditya\_Srivastava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_srivastava/32/43287_2.png) [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)\ **Post date:** [November 18, 2016, 2:33pm UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/5 "2016-11-18T14:33:58Z")
> [@Aditya\_Srivastava](#): > > No fields in all the log files will be different Sry, by this statement i meant that No, all the log files have different patterns. All are my custom application log file. Also below is the FB config file where I read from multiple path and create kafka topic for all the log files. ``` filebeat: prospectors: - paths: - /rescue/dataLogging/1.2.3.4/App__access/*.log document_type: log multiline: pattern: '^[JFMASOND]' negate: true match: after fields: type: abc14_logdata fields_under_root: true ############################ file 2 ######################### - paths: - /rescue/dataLogging/2.3.4.5/Log_data_check/*.log document_type: log fields: type: xyz14_logdata_2 fields_under_root: true ############################ file 3 ######################### - paths: - /rescue/dataLogging/3.4.5.6/TAG/*.log document_type: log fields: {type: def32_logdata} fields_under_root: true ########################## file 4################### and so on...... ##################### output ############################ output: kafka: hosts: ["192.168.0.1:9092"] topic: "%{[type]}" partition.round_robin: reachable_only: false ``` My prob if I have 100 files do I need to write all 100 paths here itself in one config file. It will be all so messy. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 18, 2016, 3:38pm UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/6 "2016-11-18T15:38:07Z")
if you're using 100 different topics for these 100 different path and without me seeing a general rule to filter on (`output.kafka.topics` setting supports conditionals), then you have to create a prospector per file/directory. As I said already, filebeat can load additional prospector configurations via `filebeat.config_dir`. No need to have one config file only. You can have one config file per prospector. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 16, 2016, 3:38pm UTC](https://discuss.elastic.co/t/include-files-inside-filebeat-config-file/66518/7 "2016-12-16T15:38:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.