Multiple input configuration

Elastic Stack Beats
1

I want to read multiple log files and send it to logstash. Below is my configuration (I just wanted to test it locally so the configuration I used is given below):

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 0

filebeat.prospectors:

- input_type: log
  enabled: true
  paths:
    - C:\data\log\eis.log
  fields: {log_type: eis}

- input_type: log
  enabled: true
  paths:
    - C:\data\log\sa.log
  fields: {log_type: sa}
 
output.file:
   path: "c:/var/"
   filename: "filebeat.log"

I see that only one of the file is getting written. How can I configure to write the other file as well ? I have updated my whole configuration file for your reference.

2

Hi @Raghuveer_SJ,

Be careful with Windows paths in filebeat configuration, backslash (\) is used to escape characters in yaml, so you need to escape these backslashes (C:\\data\\log\\eis.log), or quote the paths with single quotes ('C:\data\log\eis.log').

3

With that change no files are getting generated :slight_smile: and filebeat exited with the error :

2018-08-07T15:44:38.817+0530    ERROR   instance/beat.go:691    Exiting: Error in initing input: No paths were defined for input accessing 'filebeat.prospectors.2' (source:'filebeat.yml')
Exiting: Error in initing input: No paths were defined for input accessing 'filebeat.prospectors.2' (source:'filebeat.yml')

Can you please look into my whole configuration file that I have posted. Just to mention I am using 6.3 ELK stack. I see filebeat.inputs is mentioned not the prospectors so are they deprecated ? Anyway I see the double slash is not something that works in 6.3 version. We should use the normal path. So I still see only my first file is getting written not the second file. Kindly help.

4

Works for me in Windows with following changes in filebeat.yml
paths:
- D:\ELK\log_FileBeat*.log
- D:\ELK\AnotherLogPath*.log

5

filebeat.prospectors is deprecated in favour of filebeat.inputs yes, but both options should work by now.

What do you mean by seeing only the first file written? Files in input paths are only read.

6

But I also want to categorize them with fields so that I can make a switch in logstash grok filter can you suggest how I can do it?

7

I see only the eis.log file the other file is not there in ELK. Filebeat is harvesting only 1 file not sure why its ignoring the other file i.e. sa.log.

8

For adding fields, the fields setting you are already using should be enough.

Does the sa.log file exist in this path? Could you check the logs in case you see something related to this file?

9

The file sa.log was saved as sa.txt.log I spent a whole half a day just to figure out this ...

thanks a lot

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.