# Filebeat load balancing failing because of ssl certificate **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 11, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500 "2016-06-11T13:13:54Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 11, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/1 "2016-06-11T13:13:55Z")
I set up SSL for filebeat and logstash connections following this link. [https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04). Here i am creating a ssl key and certificate on a logstash server and copying that to filebeat clients box. And i am using that certificate to initiate a secure connection from filebeat to logstash. And its working good. But my issue here is , when i want to add one more logstash server and load balance in filebeat, its not working as the certificate is not validating new logstash instance. So i want to know how to create secure ssl based connections to two logstash instances with load balance in filebeat. ---
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 12, 2016, 6:49am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/2 "2016-06-12T06:49:20Z")
How to use multiple certificates for connecting to multiple logstash instances from filebeat using load balance?. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 13, 2016, 8:15am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/3 "2016-06-13T08:15:05Z")
Do you use a different certificate for each LS server? Can't you share the same? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 13, 2016, 11:41am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/4 "2016-06-13T11:41:37Z")
Certificates have some IP/domain name attached. By deault every LS server needs it's own certificate. That is, one either uses an CA and uses the CAs public certificate in filebeat or all LS server certificates must be available to beats. For testing (not secure), on can disable the IP/domain name check when validating certificates (not recommended in production). See tls ouptut option [insecure](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-tls.html#_insecure). ---
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 14, 2016, 5:44am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/5 "2016-06-14T05:44:34Z")
But i need to send logs to multiple logstash instances from filebeat in a secure way, I tried giving multiple certificates in certificates column, it didn't help as filebeat is shipping only to either one of them even when loadbalance : true. Following is my configuration file ```auto filebeat: prospectors: - paths: - /path/logs fields: hostip: "ipaddress" document_type: doctype output: logstash: hosts: ["host1:port","host2:port"] loadbalance : true tls: certificate_authorities: ["certificate1.crt","certificate2.crt"] logging: to_syslog: false to_files: true files: path: /var/log/filebeat name: filebeat.log rotateeverybytes: 10485760 keepfiles: 7 level: debug ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 14, 2016, 8:49am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/6 "2016-06-14T08:49:57Z")
> [@Sai\_Birada](#): > > it didn't help as filebeat is shipping only to either one of them even when loadbalance : true. Can you explain? Is it one of a) Filebeat is always shipping to host1, but never host2. After restart it's shipping to host2, but never host1? Or is it more like b) filebeat is sending to host1, pause and then send to host2. Given filebeat can connect to both logstash instances + from your configs I'd assume it being case b) . By default Filebeat has to wait for ACK from logstash before spooler can push another batch to logstash. The default [spool\_size](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_spool_size) is 2048 and the [bulk\_max\_size](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_bulk_max_size_2) in logstash output is 2048 too. Having `spool_size = Host * Worker(=1) * bulk_max_size`, will ensure the batch generated by the spooler is split into `Host * Worker` sub-batches that get push load-balanced. In this mode filebeat pushes to all logstash instances in lock-step (next batch only after all sub-batches have been ACKed). This mode keeps memory usage somewhat low. Alternatively one can enable [publish\_async](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_publish_async) (no change in spooler\_size required) to prepare more batches in spooler to be send (they are automatically load-balanced). There is some risk in current release of `publish_async` to deadlock if no logstash endpoint is available for some unspecified amount of time (happens by random). [Related issue](https://github.com/elastic/beats/issues/1829). It's already fixed and available in next release (I've no date yet, though). ---
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 14, 2016, 8:52am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/7 "2016-06-14T08:52:35Z")
It is case 1 for me, It's always pushing to host2 initially and when i stopped logstash instance in host2, and restarted it started pushing to host 1. I didit vice versa also. Basically it could push to both instances but not together. Its always pushing to either one of them. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 14, 2016, 8:57am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/8 "2016-06-14T08:57:47Z")
filebeat version? have you checked via netstat beats having 1 or 2 network connections? Anything in logs about failed send attempts? You config file uses `loadbalance : true` instead of `loadbalance: true`. Maybe there is some problem with the yaml parser. Have you checked your config containing no tabs (YAML is somewhat sensitive). ---
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 14, 2016, 11:42am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/9 "2016-06-14T11:42:17Z")
I Checked with yaml parser and its loadbalance: true and there is no config error, usually if there is some yaml issue filebeat will fail saying yaml issue right?. And also i could see through netstat that filebeat is making connections to both the hosts. However always its sending to only box. And no errors in logs. ---
**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Post date:** [June 14, 2016, 11:49am UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/10 "2016-06-14T11:49:04Z")
Thanks for response steffens, I found out the cause for this. Its just that filebeat sends logs in bulk of 2048, And the file i gave has less than that many logs, so it made a chunk and send to one of those hosts. When i gave more logs, its distributing chunks to both the hosts. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 2, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-load-balancing-failing-because-of-ssl-certificate/52500/11 "2016-07-02T13:13:57Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.