During Logstash forwarder startup throws "Failed to tls handshake EOF" Error

Elastic Stack Logstash
1

I
have a 7 node setup where we have 3 logstash, 3 elastic search and 1
logstash-forwarder.

I
have both the logstash-forwarder and load balancer (haproxy) running in the
same VM.

All
the three logstash communicates with the loadbalancer. And loadbalancer
communicates with the logstash-forwarder.
Versions:
logstash 1 - 1.5.3
logstash2 - 1.4.2
logstash3 - 1.4.3
logstashforwarder - 0.4.0

Distributor ID: CentOS
Release: 6.5
This is our setup.

The
setup was working without any issues, suddenly we are facing the issue while
starting up logstash forwarder.

Below
error is displayed while starting logstash-forwarder,

2015/09/03 12:50:33.843148 Waiting for 2 prospectors to initialise
2015/09/03 12:50:33.843329 All prospectors initialised with 0 states to persist
2015/09/03 12:50:33.843431 Loading client ssl certificate: /etc/pki/tls/certs/logstash-forwarder.crt and /etc/pki/tls/private/logstash-forwarder.key
2015/09/03 12:50:34.013314 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/09/03 12:50:34.013891 Connecting to [10.xxx.xxx.xxx]:5043 (Loadbalancer)
2015/09/03 12:50:34.014713 Failed to tls handshake with 10.xxx.xxx.xxx EOF

Please let me know if you need any further info.

Logstash-Forwarder configuration

{
"network": {
"servers": [ "Loadbalancer:5043" ],
"ssl certificate": "/etc/pki/tls/certs/logstash-forwarder.crt",
"ssl key": "/etc/pki/tls/private/logstash-forwarder.key",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 60
},
"files": [
{
"paths": [ "" ],
"fields": {
"Component": "ABC",
"logtype": "log"
}
},
{
"paths": [ "" ],
"fields": {
"Component": "ABC",
"logtype": "audit"
}
}
]
}

2

Have you looked at the logs on that host to see what is happening?

3

I checked the logs in /var/log/logstash-forwarder. The logstash-forwarder.log file is empty, while logstash-forwarder.err contains the same error lines.

Attaching:

2015/09/02 16:01:20.407785 --- options -------
2015/09/02 16:01:20.407828 config-arg: /etc/logstash-forwarder.conf
2015/09/02 16:01:20.407840 idle-timeout: 5s
2015/09/02 16:01:20.407846 spool-size: 1024
2015/09/02 16:01:20.407851 harvester-buff-size: 16384
2015/09/02 16:01:20.407855 --- flags ---------
2015/09/02 16:01:20.407859 tail (on-rotation): false
2015/09/02 16:01:20.407864 log-to-syslog: false
2015/09/02 16:01:20.407868 quiet: false
2015/09/02 16:01:20.407907 {
"network": {
"servers": [ "Loadbalancer:5043" ],
"ssl certificate": "/etc/pki/tls/certs/logstash-forwarder.crt",
"ssl key": "/etc/pki/tls/private/logstash-forwarder.key",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},

"files": [
{
"paths": [ "/home/.log" ],
"fields": {
"Component": "ABC",
"logtype": "log"
}
},
{
"paths": [ "/home/.audit" ],
"fields": {
"Component": "ABC",
"logtype": "audit"
}
}
]
}

2015/09/02 16:01:20.408631 Waiting for 2 prospectors to initialise
2015/09/02 16:01:20.409966 All prospectors initialised with 0 states to persist
2015/09/02 16:01:20.410182 Loading client ssl certificate: /etc/pki/tls/certs/logstash-forwarder.crt and /etc/pki/tls/private/logstash-forwarder.key
2015/09/02 16:01:20.585241 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/09/02 16:01:20.592656 Connecting to [10.xxx.xxx.xxx]:5043 (Loadbalancer)
2015/09/02 16:01:20.593842 Failed to tls handshake with 10.78.220.180 EOF
2015/09/02 16:01:21.594730 Connecting to [10.xxx.xxx.xxx]:5043 (Loadbalancer)
2015/09/02 16:01:21.595063 Failed to tls handshake with 10.78.220.180 EOF
2015/09/02 16:01:22.595800 Connecting to [10.xxx.xxx.xxx]:5043 (Loadbalancer)

4

Does your loadbalancer pass through the SSL to LS or does it terminate there?

5

Warkolm,

I am not sure how to verify it . is there a way i can check it ? please guide me.

Thanks,
Vino

6

It depends on how you set it up! Did you set things up like that or not?

7

i generated the keys mentioned in ssl certificate, ssl key and shared it. I did not make any other configurations