Mutual TLS FileBeat to LogStash fails with "remote error: tls: internal error"

I am trying to setup mutual authentication between FileBeat and LogStash as per https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html

I have generated certificates and keys using two different methods, by using openssl on the command line, as well as using nodepki. I get the same error.

To convert a keyfile to pkcs8 (for Logstash) I used

openssl pkcs8 -topk8 -inform PEM -outform PEM -in pki/filebeat.key -out pki/filebeat.key.pkcs8

I did not convert the keyfile to PKCS8 for FileBeat.

If I comment out ssl_certificate_authorities and ssl_verify_mode from the logstash conf, things work.
However, I do want mutual authentication so this is not good enough.

The error message I get, is remote error: tls: internal error which is reproducible via curl:

bash-4.2$ curl -v --cacert /pki/ca.pem --cert /pki/client.pem --key /pki/client.key https://logstash:5044
* About to connect() to logstash port 5044 (#0)
*   Trying 172.19.0.4...
* Connected to logstash (172.19.0.4) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /pki/ca.pem
  CApath: none
* NSS: client certificate from file
* 	subject: CN=filebeat,O=Basalt AB,L=Stockholm,ST=Uppsala,C=SE
* 	start date: May 10 13:48:11 2017 GMT
* 	expire date: May 10 13:48:11 2018 GMT
* 	common name: filebeat
* 	issuer: CN=Intermediate CA,O=ADITO Software GmbH,L=Geisenhausen,ST=Bayern,C=DE
* NSS error -12188 (SSL_ERROR_INTERNAL_ERROR_ALERT)
* Peer reports it experienced an internal error.
* Closing connection 0
curl: (35) Peer reports it experienced an internal error.

Here is my relevant configuration:

logstash.conf

input {
    beats {
        port => 5044
        ssl => true
        ssl_certificate => "/pki/server.pem"
        ssl_certificate_authorities => ["/pki/ca.pem"]
        ssl_key => "/pki/server.key"
        ssl_key_passphrase => "password123"
        ssl_verify_mode => "peer"
    }
}

filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /mnt/log/*.log

processors:
- add_cloud_metadata:

output.logstash:
  hosts: ['logstash:5044']
  ssl.certificate_authorities: ["/pki/ca.pem"]
  ssl.certificate: "/pki/client.pem"
  ssl.key: "/pki/client.key"

#logging.level: debug

How do I enable debugging of the SSL negotiation in Logstash? Nothing appears in the logs there, even though I set debugging via

PUT /_node/logging
{
"logger.logstash.inputs.beats": "DEBUG"
}

The certificates and keys seem fine:

openssl s_server -cert pki/server.pem -key pki/server.key -CAfile pki/ca.pem -www -dhparam dh.param -verify force_peer
/usr/local/opt/curl/bin/curl --cacert pki/ca.pem --cert pki/client.pem --key pki/client.key  https://logstash:4433
<HTML><BODY BGCOLOR="#ffffff">
<pre>

s_server -cert pki/server.pem -key pki/server.key -CAfile pki/ca.pem -www -dhparam dh.param -verify force_peer 
Ciphers supported in s_server binary
TLSv1/SSLv3:DHE-RSA-AES256-SHA       TLSv1/SSLv3:DHE-DSS-AES256-SHA       
TLSv1/SSLv3:AES256-SHA               TLSv1/SSLv3:EDH-RSA-DES-CBC3-SHA     
TLSv1/SSLv3:EDH-DSS-DES-CBC3-SHA     TLSv1/SSLv3:DES-CBC3-SHA             
SSLv2      :DES-CBC3-MD5             TLSv1/SSLv3:DHE-RSA-AES128-SHA       
TLSv1/SSLv3:DHE-DSS-AES128-SHA       TLSv1/SSLv3:AES128-SHA               
TLSv1/SSLv3:DHE-RSA-SEED-SHA         TLSv1/SSLv3:DHE-DSS-SEED-SHA         
TLSv1/SSLv3:SEED-SHA                 SSLv2      :RC2-CBC-MD5              
TLSv1/SSLv3:RC4-SHA                  TLSv1/SSLv3:RC4-MD5                  
SSLv2      :RC4-MD5                  TLSv1/SSLv3:EDH-RSA-DES-CBC-SHA      
TLSv1/SSLv3:EDH-DSS-DES-CBC-SHA      TLSv1/SSLv3:DES-CBC-SHA              
SSLv2      :DES-CBC-MD5              TLSv1/SSLv3:EXP-EDH-RSA-DES-CBC-SHA  
TLSv1/SSLv3:EXP-EDH-DSS-DES-CBC-SHA  TLSv1/SSLv3:EXP-DES-CBC-SHA          
TLSv1/SSLv3:EXP-RC2-CBC-MD5          SSLv2      :EXP-RC2-CBC-MD5          
TLSv1/SSLv3:EXP-RC4-MD5              SSLv2      :EXP-RC4-MD5              
---
Ciphers common between both SSL end points:
ECDHE-RSA-AES256-SHA       ECDHE-ECDSA-AES256-SHA     DHE-RSA-AES256-SHA        
DHE-DSS-AES256-SHA         ECDH-RSA-AES256-SHA        ECDH-ECDSA-AES256-SHA     
AES256-SHA                 ECDHE-RSA-AES128-SHA       ECDHE-ECDSA-AES128-SHA    
DHE-RSA-AES128-SHA         DHE-DSS-AES128-SHA         DHE-RSA-SEED-SHA          
DHE-DSS-SEED-SHA           ECDH-RSA-AES128-SHA        ECDH-ECDSA-AES128-SHA     
AES128-SHA                 SEED-SHA                   ECDHE-RSA-DES-CBC3-SHA    
ECDHE-ECDSA-DES-CBC3-SHA   EDH-RSA-DES-CBC3-SHA       EDH-DSS-DES-CBC3-SHA      
ECDH-RSA-DES-CBC3-SHA      ECDH-ECDSA-DES-CBC3-SHA    DES-CBC3-SHA
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 0120DEDB36505ADA22D1DD50C584BB3D1CE0CCF1FD2A8D913F1BC8E5B97E69A0
    Session-ID-ctx: 01000000
    Master-Key: FEAB0579630DD06952BE6F110CB43FA3AD75306C3C300C62FE937FAB73F272DC727899D2B3AC311A5B091380CCA9F779
    Key-Arg   : None
    Start Time: 1494488297
    Timeout   : 300 (sec)
    Verify return code: 26 (unsupported certificate purpose)
---
   1 items in the session cache
   0 client connects (SSL_connect())
   0 client renegotiates (SSL_connect())
   0 client connects that finished
   1 server accepts (SSL_accept())
   0 server renegotiates (SSL_accept())
   1 server accepts that finished
   0 session cache hits
   0 session cache misses
   0 session cache timeouts
   0 callback cache hits
   0 cache full overflows (128 allowed)
---
Client certificate
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4100 (0x1004)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=DE, ST=Bayern, L=Geisenhausen, O=ADITO Software GmbH, CN=Intermediate CA
        Validity
            Not Before: May 10 13:48:11 2017 GMT
            Not After : May 10 13:48:11 2018 GMT
        Subject: C=SE, ST=Uppsala, L=Stockholm, O=Basalt AB, CN=filebeat
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:ad:f1:b3:8f:d4:77:37:ef:11:17:da:c1:c2:f1:
                    71:2a:af:6c:c5:4f:d2:e5:d0:18:5a:76:d4:1f:f9:
                    78:36:f3:35:0b:2a:83:c3:a2:aa:1e:96:5c:bb:b4:
                    71:eb:a0:ce:0c:4d:bc:87:88:09:23:99:0a:07:c5:
                    cc:71:ea:34:d0:ed:f7:91:e6:f3:ab:9f:6c:ad:32:
                    d0:74:05:ab:06:88:80:2e:a5:5b:2e:f0:95:6c:08:
                    ef:97:de:cc:6e:ed:e5:fc:37:be:88:2a:b8:07:8e:
                    31:09:a2:b8:03:72:2e:d4:8e:99:c9:43:43:65:1a:
                    39:a2:70:31:ff:21:a9:2b:2e:10:9b:24:e0:c4:35:
                    ba:25:13:0e:45:0a:e0:92:cc:c5:5c:57:09:fa:98:
                    63:82:a5:7c:5b:79:91:ad:3d:61:22:b2:94:b2:27:
                    7f:59:db:af:95:7a:a9:ce:0e:5c:87:4a:4e:2e:29:
                    53:92:02:e9:af:20:24:96:25:b5:d7:f4:35:95:08:
                    da:43:8a:2a:2f:05:2f:6d:1b:17:41:56:3e:85:69:
                    4a:b8:2c:56:77:64:35:36:7b:1a:be:33:be:13:30:
                    d8:33:c2:6b:75:92:4f:e4:e3:5e:ba:90:e7:6f:36:
                    76:b3:80:e2:65:2f:2c:fc:a2:d8:26:48:93:2a:0d:
                    57:59
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                B8:B8:F8:8A:5E:1D:4A:59:E2:EE:69:68:0F:F2:A1:75:52:CD:75:90
            X509v3 Authority Key Identifier: 
                keyid:45:77:DA:6A:51:D4:23:44:3A:55:DF:E4:7D:5B:8A:82:C4:7B:C3:30
                DirName:/C=DE/ST=Bayern/L=Geisenhausen/O=ADITO Software GmbH/CN=Root CA
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Issuer Alternative Name: 
                <EMPTY>

            X509v3 CRL Distribution Points: 
                URI:http://ca.adito.local:8080/public/ca/intermediate/crl

            Authority Information Access: 
                OCSP - URI:http://ca.adito.local:2560

    Signature Algorithm: sha256WithRSAEncryption
        55:6c:0d:7d:d9:9f:30:9a:a9:88:75:10:3e:43:26:cc:f9:bb:
<snip>
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
</BODY></HTML>

Solved.

Actually, reading the OpenSSL output more carefully revealed the problem.

However, it would really be beneficial if the logs on Logstash indicated why the certificate was rejected.

In this case, the certificates were actually the culprit: They lacked a specific purpose to be used as an SSL Client.

Relevant portions of openssl.cnf:

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.