Filebeat only reporting on last prospector

Elastic Stack Beats
1

Hey guys,

I'm trying to recreate my lumberjack configuration in filebeats. My lumberjack configuration had been working great for me for a number of years! But I like the convenience of notating things in yaml way better than using json. Also I like keeping up with advances and developments from Elastic.

I've seen that in my attempt to duplicate the lumberjack config in filebeats, the config only seems to pick up the last entry which is for the php file.

In my filebeat logs I see these messages repeated over and over again:

[root@web1:~] #tail -f /var/log/filebeat/filebeat.log
2016-02-08T22:44:35-05:00 DBG  Check file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:35-05:00 DBG  Update existing file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:35-05:00 DBG  Not harvesting, file didn't change: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:37-05:00 DBG  Flushing spooler because of timemout. Events flushed: 0
2016-02-08T22:44:45-05:00 DBG  Flushing spooler because of timemout. Events flushed: 0
2016-02-08T22:44:45-05:00 DBG  Start next scan
2016-02-08T22:44:45-05:00 DBG  scan path /var/log/httpd/jf_php_error.log
2016-02-08T22:44:45-05:00 DBG  Check file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:45-05:00 DBG  Update existing file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:45-05:00 DBG  Not harvesting, file didn't change: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:52-05:00 DBG  Flushing spooler because of timemout. Events flushed: 0
2016-02-08T22:44:55-05:00 DBG  Start next scan
2016-02-08T22:44:55-05:00 DBG  scan path /var/log/httpd/jf_php_error.log
2016-02-08T22:44:55-05:00 DBG  Check file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:55-05:00 DBG  Update existing file for harvesting: /var/log/httpd/jf_php_error.log
2016-02-08T22:44:55-05:00 DBG  Not harvesting, file didn't change: /var/log/httpd/jf_php_error.log

It's only picking up the last prospector which is for the php log file. Here's my filebeat config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/httpd/jf_ref.example.com_access_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_ref_access
      paths:
        - /var/log/httpd/jf_ref.example.com_error_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_ref_error
      paths:
        - /var/log/httpd/jf_beta.example.com_access_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_beta_access
      paths:
        - /var/log/httpd/jf_beta.example.com_error_log
      document_type: apache
      fields:
         service: apache
         type: apache_beta_error
      paths:
        - /var/log/httpd/jf_dev.example.com_access_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_dev_access
      paths:
        - /var/log/httpd/jf_dev.example.com_error_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_dev_error
      paths:
        - /var/log/httpd/jf_php_error.log
      document_type: php
      input_type: log
      fields:
         service: php
         type: php
      #paths:
      #  - /var/log/*.log
      #  - /var/log/*/*.log
      #input_type: log
  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts:
     - logs.example.com:5000
    index: filebeat
    #tls:
    #  certificate_authorities:
    #    - /etc/pki/CA/certs/ca.crt
shipper:
  name: filebeat
  tags: ["example-dev", "web-tier"]
  ignore_outgoing: true
  refresh_topology_freq: 10
  logy_expire: 15

logging:
  level: debug
  # enable file rotation with default configuration
  to_files: true
  # do not log to syslog
  to_syslog: false

  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 7

What am I doing wrong so that filebeat is only picking up the php log prospector?

Thanks

2

In YAML terms the prospectors is a list of dictionaries. You only have one element in your prospectors list (the single - directly under prospectors). So add some dashes to create separate dictionaries for each prospector you want to define.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/httpd/jf_ref.example.com_access_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_ref_access
    -
      paths:
        - /var/log/httpd/jf_ref.example.com_error_log
      document_type: apache
      input_type: log
      fields:
         service: apache
         type: apache_ref_error
    # You should also be able to put the dash on the same line as paths. 
    - paths:
        - /var/log/httpd/jf_beta.example.com_access_log
      document_type: apache
    ...
3

Thanks for the advice! That did it. Filebeat works fine now. :smile: