Filebeat: Reading Prospector Config from Directory

Elastic Stack Beats
1

Hi,
I have been reading some test logs successfully with filebeat. However, when I move the prospector config out into an external directory it fails. Fielbeat sees the directory and the file:

2015-12-22T11:48:11Z INFO Init Beat: filebeat; Version: 1.0.1
2015-12-22T11:48:11Z INFO Additional config files are fetched from: /etc/filebeat/conf.d
2015-12-22T11:48:11Z INFO Additional configs loaded from: /etc/filebeat/conf.d/cassandra-collect.yml
2015-12-22T11:48:11Z INFO filebeat sucessfully setup. Start running.
2015-12-22T11:48:11Z INFO Registry file set to: /var/lib/filebeat/registry
2015-12-22T11:48:11Z INFO Loading registrar data from /var/lib/filebeat/registry
2015-12-22T11:48:11Z DBG Set idleTimeoutDuration to 5s
2015-12-22T11:48:11Z DBG File Configs: []
2015-12-22T11:48:11Z DBG Set ignore_older duration to 24h0m0s
2015-12-22T11:48:11Z DBG Set scan_frequency duration to 10s
2015-12-22T11:48:11Z DBG Set backoff duration to 1s
2015-12-22T11:48:11Z DBG Set max_backoff duration to 10s
2015-12-22T11:48:11Z DBG Set partial_line_waiting duration to 5s
2015-12-22T11:48:11Z DBG Waiting for 1 prospectors to initialise
2015-12-22T11:48:11Z DBG No pending prospectors. Finishing setup
2015-12-22T11:48:11Z INFO All prospectors initialised with 1 states to persist
2015-12-22T11:48:11Z INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2015-12-22T11:48:11Z INFO Starting Registrar
2015-12-22T11:48:11Z INFO Start sending events to output

...however, nothing seems to be getting read:

2015-12-22T11:48:51Z DBG Start next scan
2015-12-22T11:48:56Z DBG Flushing spooler because of timemout. Events flushed: 0

...even though the input log is being written to.

External prospector config looks like this:

paths:
  - /var/log/cassandra/system.log
input_type: log
fields:
  cluster_name: test_cluster
  datacenter: dc1
document_type: dse
scan_frequency: 5s

Regards,
David

2

Can you share you complate filebeat.yml file? What's in /etc/filebeat/conf.d ? The additional config files must end with ".yml" .

I'm missing the log output (INFO level): Additional config files loaded from.

3

Name of prospector config file is :

/etc/filebeat/conf.d/cassandra-collect.yml

Filebeat config is:

filebeat:
  prospectors:
      -
  registry_file: /var/lib/filebeat/registry
  config_dir: /etc/filebeat/conf.d
  
output:
  logstash:
    hosts: ["logstash_ip:5044"]
  
shipper:

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 10
  level: debug

Regards,
David

4

remove 'prospector' from filebeat config.

Update cassandra-collect.yml to:

filebeat:
  prospectors:
    - 
      paths:
        - /var/log/cassandra/system.log
      input_type: log
      fields:
        cluster_name: test_cluster
        datacenter: dc1
      document_type: dse
      scan_frequency: 5s