# Filebeat: Reading Prospector Config from Directory

**URL:** <https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752>\
**Category:** Beats\
**Created:** [December 22, 2015, 12:07pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752 "2015-12-22T12:07:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [December 22, 2015, 12:07pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752/1 "2015-12-22T12:07:08Z")

</div>

Hi,  
I have been reading some test logs successfully with filebeat. However, when I move the prospector config out into an external directory it fails. Fielbeat sees the directory and the file:

2015-12-22T11:48:11Z INFO Init Beat: filebeat; Version: 1.0.1  
2015-12-22T11:48:11Z INFO Additional config files are fetched from: /etc/filebeat/conf.d  
2015-12-22T11:48:11Z INFO Additional configs loaded from: /etc/filebeat/conf.d/cassandra-collect.yml  
2015-12-22T11:48:11Z INFO filebeat sucessfully setup. Start running.  
2015-12-22T11:48:11Z INFO Registry file set to: /var/lib/filebeat/registry  
2015-12-22T11:48:11Z INFO Loading registrar data from /var/lib/filebeat/registry  
2015-12-22T11:48:11Z DBG Set idleTimeoutDuration to 5s  
2015-12-22T11:48:11Z DBG File Configs: []  
2015-12-22T11:48:11Z DBG Set ignore\_older duration to 24h0m0s  
2015-12-22T11:48:11Z DBG Set scan\_frequency duration to 10s  
2015-12-22T11:48:11Z DBG Set backoff duration to 1s  
2015-12-22T11:48:11Z DBG Set max\_backoff duration to 10s  
2015-12-22T11:48:11Z DBG Set partial\_line\_waiting duration to 5s  
2015-12-22T11:48:11Z DBG Waiting for 1 prospectors to initialise  
2015-12-22T11:48:11Z DBG No pending prospectors. Finishing setup  
2015-12-22T11:48:11Z INFO All prospectors initialised with 1 states to persist  
2015-12-22T11:48:11Z INFO Starting spooler: spool\_size: 1024; idle\_timeout: 5s  
2015-12-22T11:48:11Z INFO Starting Registrar  
2015-12-22T11:48:11Z INFO Start sending events to output

...however, nothing seems to be getting read:

2015-12-22T11:48:51Z DBG Start next scan  
2015-12-22T11:48:56Z DBG Flushing spooler because of timemout. Events flushed: 0

...even though the input log is being written to.

External prospector config looks like this:

```
paths:
  - /var/log/cassandra/system.log
input_type: log
fields:
  cluster_name: test_cluster
  datacenter: dc1
document_type: dse
scan_frequency: 5s

```

Regards,  
David

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 22, 2015, 3:25pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752/2 "2015-12-22T15:25:48Z")

</div>

Can you share you complate filebeat.yml file? What's in /etc/filebeat/conf.d ? The additional config files must end with ".yml" .

I'm missing the log output (INFO level): Additional config files loaded from.

---

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [December 22, 2015, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752/3 "2015-12-22T15:37:16Z")

</div>

Name of prospector config file is :

/etc/filebeat/conf.d/cassandra-collect.yml

Filebeat config is:

```
filebeat:
  prospectors:
      -
  registry_file: /var/lib/filebeat/registry
  config_dir: /etc/filebeat/conf.d
  
output:
  logstash:
    hosts: ["logstash_ip:5044"]
  
shipper:

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat.log
    rotateeverybytes: 10485760
    keepfiles: 10
  level: debug

```

Regards,  
David

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 23, 2015, 3:50pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752/4 "2015-12-23T15:50:49Z")

</div>

remove 'prospector' from filebeat config.

Update cassandra-collect.yml to:

```
filebeat:
  prospectors:
    - 
      paths:
        - /var/log/cassandra/system.log
      input_type: log
      fields:
        cluster_name: test_cluster
        datacenter: dc1
      document_type: dse
      scan_frequency: 5s
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:57pm UTC](https://discuss.elastic.co/t/filebeat-reading-prospector-config-from-directory/37752/5 "2017-07-05T21:57:17Z")

</div>