ERR SSL client failed to connect with:

Elastic Stack Logstash
1

I think this is a Logstash config issue hence posting in this category since I have multiple nodes reporting same message.

When starting filebeat on a node I get the above message ending with ":5044: getsockopt: connection refused"

Any ideas?

Thanks

Paul

2

can you provide the conf for filebeat input on LS and your filebeat.yml config for LS connection.

3

Input with cert values removed

input {
beats {
port => 5044
ssl => true
ssl_certificate => ""
ssl_key => ""
}
}

the filebeat config file has been set to the test setting for TLS for now. No ES just LS configured.

Do you think it is a TLS problem?

4

it doesn't sound like an ssl/tls issue as the error message would mention it. Are there any firewall, iptables rules blocking it from accessing LS. Can you telnet from the filebeat host to the LS host on port 5044? you can use curl to test ssl connection to logstash to see if its working? curl -v --cacert ca.crt https://myLShost:5044

5

Good suggestion. I cant open a telnet connection to 5044!

I went back to the beginning on this one. Checked all config. Still cannot find anything!

6

if you can't telnet to the host:port and logstash is running/listening on that port, check the iptables on the logstash host. Check the messages log to see if there is any message about connection being blocked. Use netstat on the logstash host to make sure the logstash java process is listening on port 5044. If you can't reach the host and port via telnet, there is something blocking it. You need to see what is blocking connection from reaching your logstash host on that port. You may need to use tcpdump/wireshark to help troubleshoot the network connectivity issue if its not something as simple as iptables rules blocking that port or iptables rules not open for that port.

7

Using netstat -atn I can see that 5044 is not listening and this is odd because I haven't changed it or the security on the serve. Just incase I put selinux in permissive mode just, rebooted and stopped IPtables (this is not a live system) but no joy. I'm wondering if the issue is with some corruption in Logstash?

8

trying running logstash process manually as root. in one terminal run:

/opt/logstash/bin/logstash -f /etc/logstash/conf.d

in a second terminal see if you can see the port being listened on with netstat. Then try it again as the logstash user via sudo.

9

Thanks, very useful return has led me to an incorrect conf file within the directory. Needs some trial and error around the advice found https://www.elastic.co/blog/elasticsearch-and-siem-implementing-host-portscan-detection.

Will update once I find the cause but SSL error match on something that I have been trying to do that the configtest has passed but your suggestion highlighted.

Update: blocked
All config cut down to minimum, local security checked, 5044 not listed in listening mode (as root).
Changed my logstash input file from beats to tcp for 5044 and a connection is accepted. Change it back to beats and refused.

Updated logstash and beats plugin, reboot and still nothing. Wondering if the beat component on my ELK stack is the issue?? Mystery continues...

10

Let take a step back. First lets get logstash listening on 5044 using beats with no certs. So set the logstash conf.d for beats to something like this:

01-filebeat-input.conf: 
input {
   beats {
      port => 5044
    }
}

and outupt to stdout

 30-stdout-output.con: 
     output {
           stdout { codec => rubydebug }
    }

start logstash. Check to see if logstash is listening on 5044 for beats input without ssl. if that doesn't work, then i would consider reinstalling the logstash plugins or reinstalling the logstash package.

11

I had this exact same error! The SSL client failed to connect is actually rather misleading. at least in my case it was down to the indentation of the yml on the outbound beat, i.e. the node that I was shipping logs from.... I know this sounds crazy, but it was true, had me baffled for ages.... try using this on the node that you are trying to send logs from! See below:

filebeat:
prospectors:
-
paths:
- /usr/local/apache-servicemix-7.0.0.M1/data/log/*.log

  input_type: log

  document_type: servicemixlog

registry_file: /var/lib/filebeat/registry

output:
logstash:
hosts: ["xx.x.x.xxx:5044"]
bulk_max_size: 1024

tls:
  certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
files:
rotateeverybytes: 10485760 # = 10MB

Let me know how you get on!

P.s replace the irrelevant bits that I left in there!

12

Thanks Luka and Michael, both suggestions tried but not yet fixed. I could be suffering from yaml nitrate!! As you have found Luka so many issues caused by indents but not today.

So I'm going to try reinstalling Logstash will keep you posted.

ps. If anyone knows a good yaml editor that can sit with assoicated app demanding the use of yaml please let me know.

UPDATE: After reinstalling Logstash I was able to connect to 5044. I also reissued the TLS cert which caused problems by the introduction of a single "+" character at the beginning of my domain name. I noticed this when loading onto a client to check. I suppose the answer to full deployment is centralise through a CA, especially for Beating Windows clients. At least I know a lot more about Logstash that I used to.