Can filebeat send to different LS servers based on hostname criteria

Elastic Stack Beats
1

Hello,

I'm wondering if filebeat can be configured with many output.logstash hosts but would send the data only to one LS acording to a condition.

For instance, I have 4 machines (F1,F2,F3,F4) owning the same configuration of filebeat and 2 LS servers (L1, L2). I would like to send files from F1,F2,F3 to L1 and from F4 to L2 :

output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Logstash hosts
  hosts: ["L1"]
  when:
    or
	  - equals:
        beat.hostname: "F1"
	  - equals:
        beat.hostname: "F2"
	  - equals:
        beat.hostname: "F3"		

  hosts: ["L2"]
  when:
    equals:
      beat.hostname: "F4"

Else I need to own differents configurations on each F machine :frowning:

Is it possible ?

PS : limitation might be the same as Can filebeat be configured to send to different ES or logstash servers?

Best regards

2

PS : limitation might be the same as Can filebeat be configured to send to different ES or logstash servers?

Right. Filebeat does not support event routing.

You could provide the filebeat configuration as a template and have your provisioning system (or some custom startup script) create the final per machine configuration from your template.

One can also use variables like this:

output.logstash:
  hosts: ["${LOGSTASH_HOST}"]

Variables can be either set via:

  • environment variables
  • another configuration file and start filebeat with multiple config files: (e.g. filebeat -c main.yml -c vars.yml )
  • add variable to startup command: (e.g. filebeat -E LOGSTASH_HOST=...)
  • having provisioning, just append variables to end of the config file like (echo LOGSTASH_HOST: ... >> /etc/filebeat/filebeat.yml)
3

Thank you for your reply Steffen .

If it can help somebody, best solution I found is :

  • create a new yml file named /appli/projects/filebeat-6.0.1-linux-x86_64/envs/filebeat_MYHOSTNAME.yml owning the configuration with LOGSTASH_HOST=...
  • change /etc/systemd/system/filebeat.service by adding -c /appli/projects/filebeat-6.0.1-linux-x86_64/envs/filebeat%H.yml
  • modify my filebeat.yml to use the variable : hosts: ["${LOGSTASH_HOST}"]

So, I'm now able to separate common configuration and server configuration :wink:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.