Problem to define multiples propsector with filebeat

Elastic Stack Beats
1

Hi everybody,
I am new to filebeat. I do have some issues to properly format filebeat to use multiples prospector. My goal is to separate my file type with a separate prospector to add selector fields used after in my logstash instance to parse the row properly and then index in the proper index.

My prospector block look like the following:

filebeat:
prospectors:
     - 
       paths:
            - "/var/log/messages*"
       encoding: plain
       input_type: log
       document_type: log
       ignore_older: 240h
       fields:
            log_type: "messages"
            index_name: "my_index"
    -
   ...

I am presently getting an error if I put more than one prospector in the same filebeat configuration file. so currently, I do start manually one filebeat instance per file type. What is the proper way to achieve what I want.

2

Since YAML is sensitive to leading whitespace, please format your configuration snippet as code so we can see exactly what you're using.

I am presently getting an error if I put more than one prospector in the same filebeat configuration file.

What did you try and what error message did you get?

3

When you mentioned that the space are important with the yaml configuration file, I did watch closely how your file was built. Correct me if I am wrong: The paths argument has to be the next line and the next column, no extra space. path,encoding, input_type, document_type, ignore_older and fields all aligned starting at the same column. After, I did those changes, I can't clone my error and it seems you fix my problem.

4

There doesn't have to be a linebreak before the hyphen and "path", i.e. this is fine too:

filebeat:
  prospectors:
    - paths:
        - "/var/log/messages*"
      encoding: plain
      input_type: log
      document_type: log
      ignore_older: 240h
      fields:
        log_type: "messages"
        index_name: "my_index"
5

I faced with same problem, with the 5.3 version of Filebeat. ELK stack is installed on CentOS 7.3 x64 server.

This is part of my filebeat.yml file:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/httpd/access_log
document_type: accesslog1

- input_type: log
  paths:
    - /var/log/httpd/error_log
document_type: errorlog

When I try to start filebeat service, service is failing to start and getting this error:

root@filebeatcli1(~)$systemctl start filebeat.service
[15:19] root@filebeatcli1(~)$systemctl status filebeat.service
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2017-04-12 15:19:55 CEST; 3s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Process: 2774 ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat (code=exited, status=1/FAILURE)
Main PID: 2774 (code=exited, status=1/FAILURE)

Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: filebeat.service: main process exite...E
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: Unit filebeat.service entered failed....
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: filebeat.service failed.
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: filebeat.service holdoff time over, ....
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: start request repeated too quickly f...e
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: Failed to start filebeat.
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: Unit filebeat.service entered failed....
Apr 12 15:19:55 filebeatcli1.bsmain.local systemd[1]: filebeat.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

If I comment any of prospectors, service is starting correctly. They cannot work together Example:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/httpd/access_log
document_type: accesslog1

#- input_type: log
#  paths:
#    - /var/log/httpd/error_log
#document_type: errorlog

Please help me if somebody knows, what I have done wrong

Thank you in advance

6

I got it working.

You were right, there likely was a problem with yaml syntax. Probably I had some spaces that I could not see, or something similar

I've done following:
Renamed my filebeat.yml to filebeat.yml.old. File filebeat.full.yml renamed to filebeat.yml. In a new config file, in Filebeat prospectors carefuly entered following configuration:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/httpd/access_log
  document_type: accesslog

- input_type: log
  paths:
    - /var/log/httpd/error_log
  document_type: errorlog

This configuration worked, I am now able to filter documents in Logstash using document type, for different log formats.

7

There is a problem in 5.3 when there is an invalid prospector, it can happen that filebeat panics instead of shutting down properly. Here is the potential fix for it: https://github.com/elastic/beats/pull/4037