Difficulties with external configs in FileBeat

Elastic Stack Beats
1

I have a working filebeat (6.2.2) configuration but am missing the ability to change the config (during development) and have filebeat pick up the changes without being restarted. As I understand it, this is only possible with external configuration files but I have not been able to find any example of how such .yml files should look - at least only conflicting examples and no working examples.

Just copying my - type sections into an external .yml file has not done the trick.

my filebeat.yml :

filebeat.prospectors:
  # -type sections were here...
  enabled: true
  path: D:\elk\filebeat\config\*_dev.yml

D:\elk\filebeat\config\filebeat_access_dev.yml :

# Parse apache tomcat "access" log
- type: log
  enabled: true
  paths:
	- D:\tmp\log\access.*.log
  # We exclude redirects, pings and health checks, JavaScripts and CSS
  exclude_lines: [' 302 ', 'UptimeRobot', 'healthCheck', '_ui/', '\.js', '\.css']
  fields:
	source: access
	server: app02
  scan_frequency: 3s

Needless to say, filebeat does not monitor changes to files in D:\tmp\log\access.*.log. It also does not log anything relating to D:\elk\filebeat\config\*_dev.yml and only mentions scanning for modules in ./modules.d/*.yml

2018-04-20T13:20:10.683+0200 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 0

Filebeat also does not seem to raise any error or info logs (even using -v or -d *) if the .yml files are incorrectly structured or the the scan path does not exist!

2

Filebeat uses YAML for it's configuration. A : e.g. might not be parsed into a string, but a dictionary. On windows always put paths into single quotes (single quotes don't interpret the \ as escape character): 'D:\tmp\log\access.*.log.

Have you checked cases of paths? Unfortunately the glob used by filebeat is case-sensitive.

According to filebeat docs on external loading, the external loading is configured via filebeat.config.prospectors, not filebeat.prospectors.

3

I've not had a problem before with unquoted paths e.g.:

path.data: D:\elk\filebeat\data_dev

However you are right about filebeat.config.prospectors - that has solved it for me thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.