Filebeat config prospectors not working in 6.0

Elastic Stack Beats
1

Hey everybody,
So I upgraded filebeat to version 6.0.0 and I saw this message in the logs:

WARN DEPRECATED: config_dir is deprecated. Use filebeat.config.prospectors instead. Will be removed in version: 7.0.0

So I went in the docs and found this:
https://www.elastic.co/guide/en/beats/filebeat/6.x/filebeat-configuration-reloading.html#load-prospector-config

Which says I need to add something like this to my config instead:

filebeat.config.prospectors:
  enabled: true
  path: configs/*.yml

However, when I did this and restarted filebeat it did not load any of my yml files in that directory and gives me no errors? What am I doing wrong?

2

Can you past a sample of the YML files in config/?

The syntax is a bit different in the sense that when using the new option, the fragment YAML should contain an array of prospectors, while in the old one, the filebeat.prospectors "header" was needed.

3

Sure thing:

filebeat.prospectors:

- type: log

  paths:
    - /opt/jboss/log/px-web-jsp.json

  json.keys_under_root: true
  fields_under_root: true
  fields:
    type: px-web-server


  scan_frequency: 10s
  ignore_older: 24h
  close_removed: true
  clean_removed: true
  close_inactive: 5m
  clean_inactive: 30h

I tried commenting out the "filebeat.prospectors:" part but it didn't seem to do anything

4

The syntax should be good (if you take out the filebeat.prospectors: part). Can you run with -d "*" and paste the output?

5

Hm, so I just redid all the steps again and I think it is working? I see this in the log:

2017-11-17T12:14:51-05:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 0
2017-11-17T12:14:51-05:00 INFO Config reloader started
2017-11-17T12:14:51-05:00 INFO Starting 2 runners ...
2017-11-17T12:14:51-05:00 INFO Starting prospector of type: log; id: 10730498759109995509
2017-11-17T12:14:51-05:00 INFO Starting prospector of type: log; id: 15532996734736054351
2017-11-17T12:14:51-05:00 INFO Loading of config files completed.
2017-11-17T12:14:51-05:00 INFO Harvester started for file: /opt/jboss/px-web-jsp/log/localhost_access_log.log
2017-11-17T12:14:51-05:00 INFO Harvester started for file: /opt/jboss/px-web-jsp/log/px-web-jsp.json

So... it looks like there are 0 prospectors but it is harvesting the logs? Not sure how that is possible. As far as what fixed it I'm not sure, the only thing I did difference was remove the "filebeat.prospectors" line from the yml file instead of commenting it out

6

I have the exact same problem in v6.0.0

.\filebeat.exe --version
filebeat version 6.0.0 (amd64), libbeat 6.0.0

This works:

filebeat.config_dir: 'C:\Program files\filebeat\conf.d'

And this does not:

filebeat.config.prospectors:
  enabled: true
  path: conf.d/*.yml

Example prospector in conf.d (test.yaml):

filebeat:
  prospectors:
    -
      paths:
        - C:\ProgramData\example\logs\*.log
      fields:
        type: debuginclude
      fields_under_root: true
      encoding: windows-1252
      multiline:
        pattern: ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})
        negate: true
        match: after

@tudor Do you need anything else to continue debugging?

7

You have to remove the filebeat.prospector namespace. The files in conf.d are supposed to contain lists of prospector configurations only. An Example is given in the docs.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.