Multiple config file logstash pipelines

[kingmario]

Hey,

I am setting up ELK with filebeats. I created some pipelines (apache, ssh..) in logstash. In my webserver, I setting up filebeat.

I have multiple config file that I moved in a created directory in "/etc/filebeat/conf.d/*" :

ssh.yml :

filebeat.prospectors:                                                  
                                                                       
- type: log                                                                  
  paths:                                                                           
    - "/var/log/auth.log"
  fields:
    log_type: ssh
    pipeline: ssh
  enabled: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
    hosts: ["192.168.10.1:5443"]
    ssl.certificate_authorities: ["/etc/ssl/logstash-forwarder.crt"]

apache.yml :

filebeat.prospectors:                                                  
                                                                       
- type: log                                                                  
  paths:                                                                           
    - "/home/user/http/logs/access.log"
  fields:
    log_type: http
    pipeline: apache
  enabled: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
    hosts: ["192.168.10.1:5444"]
    ssl.certificate_authorities: ["/etc/ssl/logstash-forwarder.crt"]

How I can say to filebeat : "Go in conf.d directory and launch the config files" ?

I tested this config in filebeat.yml :

filebeat.config.prospectors:
  enabled: true
  path: /etc/filebeat/conf.d/*.yml

but in the logs, he said to me that "Please define on under the output section".

Thank you !

[adrisr (Adrian Serrano)]

You can try one of this options:

• Place your configuration files in /etc/filebeat/modules.d/
• Modify module loader to also load files from /etc/filebeat/conf.d/:

filebeat.config.modules:
  path: ${path.config}/*.d/*.yml

[kingmario]

Thank you,

but it doesn't work, because in the filebeat configuration, I don't precise an output (they are precised in the conf.d modules).

Unfortunately I understood that filebeat can't send the output to differents instances (on *:5443, *:5444...). I think that I need to launch multiples instances of filebeat or need to use a redis or kafka server to send all the logs to my logstash pipelines.