Filebeat not able to ship logs for multiple paths

Elastic Stack Beats
1

I have the below configuration for filebeat :

--- 
filebeat.config.modules: 
  path: "${path.config}/modules.d/*.yml"
  reload.enabled: false
filebeat.inputs: 
  - 
enabled: true
paths: 
  - '/builds/v2018/jenkins/jobs/ELK_CI-ID/builds/*/log'
  - '/builds/v2018/jenkins/jobs/ELK_TEST/builds/*/log'
type: log
output.logstash: 
  hosts: 
- "10.50.xx.6x:5044"
  ssl.certificate_authorities: 
- /etc/filebeat/security2/logstash.crt
setup.kibana: ~
setup.template.settings: 
  index.number_of_shards: 3

As can be seen in the above configuration , the number of paths are 2 here . In the filebeat logs , I can see the harvester getting started .

The problem is that I have 30 paths in total - all unique paths. When I add all the 30 paths , I dont see any logs in kibana. The indices size doesnot increment . However , the filebeat status shows as active and running. When I add all the paths , I dont see logs from the above two paths as well . When I had only 2 paths , the logs were visible perfectly.Any idea on what I might be doing wrong here ?

2

Hello @mrashid,

Lets try a few questions first to get a bit more details:

  • Is there any errors in the log concerning the harvester?
  • What version of filebeat you are running?
  • How many files we are trying to monitor total in all the directories?
  • Do you have directories in directories / many level deep?
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.