Multiple Logstash output with different SSL signing authorities, certificates and keys

[Platypus]

Hello,

I need Filebeat to send the same logs to two different logstash endpoints with differents signing authorities, certificates and keys. I have test them separetly and the connexion with both endpoints works fine. But, once I try them at the same time filebeat only sends the logs to one of the endpoints.

First I tried the load balancing option:

output.logstash:
   hosts: ["end_point1:5044", "end_point2:5044"]
   loadbalance : true
   ssl.certificate_authorities: ["/etc/ssl/certs/end_point1.cer", /etc/ssl/certs/end_point2.cer"]
   ssl.certificate: "/etc/ssl/certs/end_point1.cer"
   ssl.key: "/etc/ssl/certs/end_point1.key"
   ssl.certificate: "/etc/ssl/certs/end_point2.cer"
   ssl.key: "/etc/ssl/certs/end_point2.key"

I tried also to put both keys and certificates in a list as for certificate_authorities but I got a invalid type error. And then I tried to define two ouputs:

output.logstash:
  hosts: ["end_point1:5044"]
  ssl.certificate_authorities: ["/etc/ssl/certs/end_point1.cer"]
  ssl.certificate: "/etc/ssl/certs/end_point1.cer"
  ssl.key: "/etc/ssl/certs/end_point1.key"

output.logstash:
  hosts: ["end_point2:5044"]
  ssl.certificate_authorities: ["/etc/ssl/certs/end_point2.cer"]
  ssl.certificate: "/etc/ssl/certs/end_point2.cer"
  ssl.key: "/etc/ssl/certs/end_point2.key"

But none of this worked...

Any ideas?

note: I have tried loadbalancing without using ssl and it also worked

[steffens (Steffen Siering)]

This kind of use-case has been discussed here: https://github.com/elastic/beats/issues/1035

filebeats purpose is to ship logs off, of one server. Implicit coupling between to LS instances via filebeat can lead to troubles in doing so. It's recommended to use an intermedia queuing service (LS with persistent queues, redis, kafka), to ship logs and decouple the downstream systems.

[Platypus]

Thank you for your reply but isn't there any other way of doing this without adding a queuing system? This is only a temporary solution while we migrate from one platform to another and I will prefer not adding more complexity.

[steffens (Steffen Siering)]

You can have two filebeat instances run (each with it's own registry file). As every filebeat instance is processing the same file, there will be (almost) no coupling between the systems you push to.