Prospectors from config_dir don't produce any harvesters

Elastic Stack Beats
1

Hello,

I just started with filebeat and I encountered a problem.
I want to feed my filebeat service with multiple config files and so I created a directory /etc/filebeat.d and copied the /etc/filebeat/filebeat.yml to /etc/filebeat.d/myapp.yml and edited it to point to a /tmplog/*.log pattern.
I configured the config_dir in /etc/filebeat/filebeat.yml to point to /etc/filebeat.d.

I create a test logfile

echo "dummy message" >> /tmplog/apa.log

When I start the filebeat service I see this in the syslog:

...
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: beat.go:107: Init Beat: filebeat; Version: 1.0.0
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: config.go:129: Additional config files are fetched from: /etc/filebeat.d/
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: config.go:107: Additional configs loaded from: /etc/filebeat.d/myapp.yml
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: beat.go:133: filebeat sucessfully setup. Start running.
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: registrar.go:66: Registry file set to: /var/lib/filebeat/registry
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: registrar.go:76: Loading registrar data from /var/lib/filebeat/registry
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/auth.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: spooler.go:77: Starting spooler: spool_size: 1024; idle_timeout: 5s
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/bootstrap.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/boot.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/cloud-init.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/cloud-init-output.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/iptables.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/daemon.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/mail.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/kern.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/yum.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: log.go:62: Harvester started for file: /var/log/user.log
Dec 16 18:38:58 localhost /usr/bin/filebeat[36889]: crawler.go:78: All prospectors initialised with 11 states to persist
...

So no mention of any harvester picking up the log file from the pattern defined by myapp.yml.
But there are no warnings or errors in the log that indicate if I have a bad configuration file.

If I just add the same pattern as a second path to the prospector in /etc/filebeat/filebeat.yml it works without any problems.

I have enabled debug level logging but found nothing of any use.

Can someone explain how I should continue to troubleshoot this?

Best Regards
/Thomas

2

Can you please post your configuration files.

3

Oops,

Seems I had made some configuration error in the /etc/filebeat.d/myapp.yml file. I'm not sure what.
I recopied the original filebeat.yml file and very carefully only changed the path setting, and now it worked.

So I won't bother posting my configuration. Thanks for the swift reply though!

I really wish for some feedback in the logs when there are errors in a configuration file and not just the kind of happy log output I got in this case.

Best Regards
/Thomas