Force_peer/peer option fails to validate client certificate

[Akshaykapoor]

This is the following i tried,

1. Had server cert enabled on logstash with sl_verify_mode: none.
   This worked fine and and filebeat was able to connect to server and send logs.

2. Now, when i enable ssl_verify_mode: force_peer on logstash. Filebeat does not send connect to logstash and gives the following error.

2016-09-06T22:29:07Z ERR SSL client failed to connect with: EOF
2016-09-06T22:29:07Z INFO Connecting error publishing events (retrying): EOF
2016-09-06T22:29:07Z INFO send fail

following file is the config file in filebeat and logstash

tls:
    certificate_authorities: ["/etc/pki/tls/ca-root/prod-chain.cert.pem"]
    certificate: "/etc/pki/tls/certs/t1-pod.cert.pem"
    certificate_key: "/etc/pki/tls/certs/t1-pod.key.pem"

  beats {
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/s-pod.cert.pem"
    ssl_certificate_authorities => "/etc/pki/tls/ca-cert/prod-chain.cert.pem"
    ssl_verify_mode => force_peer
    ssl_key => "/etc/pki/tls/certs/s-pod.key.pem"
    port => 5043
  }

[andrewkroh]

Can you please post more of your Filebeat configuration so that we can verify you have indented the configuration correctly.

Are you able to connect to Logstash using curl? On success you should get a response like "curl: (52) Empty reply from server" since Logstash isn't an actual HTTP server.

curl -v --cacert ca.crt --cert client.crt https://server:5043

[steffens (Steffen Siering)]

looks like you're using intermediate CAs.

Which versions are you using? Anything in logstash logs (have you tried logstash in debug mode)?

[Akshaykapoor]

When using curl, i get the same error as i get in the filebeat logs. which is an SSL connect error.

Below is the error when using curl.
* NSS error -5938
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error

filebeat config i'm using for tls is as follows,

    output:
      logstash:
        timeout: 15
        hosts: ['x.x.x.x:5043']
        tls:
          certificate_authorities: ["/etc/pki/tls/ca-root/no-intermediate-ca.pem"]
          certificate: "/etc/pki/tls/certs/t-pod-no-intermediate.cert.pem"
          certificate_key: "/etc/pki/tls/certs/t-pod-no-intermediate.key.pem"

So in my logstash configuration when i disable ssl_verify_mode and ssl_certificate_authorities, the same certificates are valid and i can see the logs coming in.

Basically when enabling client certificate validation, this doesn't work (Initially i had intermediate CA's, removing that didn't help either). Does it mean that filebeat can validate the server certificate but logstash cannot validate when enabling client certificate check.

environment details,
1. filebeat version 1.2.2 (amd64)
2. logstash 2.3.1
3. CentOS 6.6

About the certificates:
Both client and server have certificates with different CN names and signed with the same root_ca

[Akshaykapoor]

@andrewkroh can you help with this.. Any ideas ? or am i missing something

Thanks in advance

[andrewkroh]

@Akshaykapoor

So in my logstash configuration when i disable ssl_verify_mode and ssl_certificate_authorities, the same certificates are valid and i can see the logs coming in.

If it works when you disable the validation of the client certificates then there is probably a problem with the certificates used by Filebeat to authenticate itself. You can use openssl s_client to negotiate a connection to Logstash. This will probably give you some more details on the connection issues.

openssl s_client -connect logstash:5043 -CAfile no-intermediate-ca.pem -cert filebeat-client.crt -key filebeat-client.key -showcerts

You may also want to add -debug.