Filebeat logging certificate error even when connection can be established

[widhalmt (Thomas Widhalm)]

Hi,

I searched the board and other sources but all I found didn't match my case or did not receive a reply. So sorry if I missed something but at least I tried.

I have beats that connect to two Logstash instances. I see logs coming through and everything is more or less as it should. My problem is that sometimes I see the following message in Filebeats logs:

2022-02-22T14:43:16.950Z ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to failover(backoff(async(tcp://logstash01.example.com:5044)),backoff(async(tcp://logstash02.example.com:5044))): x509: certificate is valid for logstash01, logstash01.example.com, not logstash02.example.com

and an accompanying log entry in Logstash about connection loss due to bad_certificate.

I'm quite puzzled why this is happening. I mean, the log entry is right. The certificate on logstash01 is not vaild for logstash02 and vice versa. But that's pretty much as it's itended to be, right?

I tried openssl s_client and I can connect to the Logstash hosts without problems when I use the same ca.crt file I use in filebeat.

I would know where to search when there wouldn't be any connection possible. But since it's working most of the time and just logging the error from time to time I'm lost.

Could you give me a hint? We thought about building certificates that are valid for both Logstash hosts but I don't think that's more than a workaround.

[widhalmt (Thomas Widhalm)]

Update: It looks like this only happens when you don't set loadbalance: true. I enabled loadbalancing and now it's working like it should.

I still see connection resets but that might be due to timeouts in the Logstash beats input. The host I'm testing with has a rather low events per second rate.

What concerns me a lot is that this effect might actually keep Filebeat from failing over to the other host. I did some tests and filebeat stopped working as soon as I deactivated the Logstash instance it was initially connected to. After a restart of Filebeat it connected to other Logstash host and events were sent like expected. But if I turned up this Logstash instance and brought back on the other one Filebeat wouldn't fail back either.

In short: With loadbalance: true it works like it should. Without loadbalancing we see errors and failover won't work.