Auditbeat to Logstash SSL errors with "Not an SSL/TLS record"

My goal is connect Auditbeat to Logstash and encrypt the communication. I am not worried about having Logstash verify the client. I do have client certificates on some servers, but I would prefer not to use them, as I don't want to create client certificates on all servers.

I am running my own CA, with self generated certificates.

Logstash version 8.11.0
Auditbeat version 8.11.1

Here is my Logstash input for Auditbeat:

input {
    beats {
        id => "linux_auditbeat"
        port => 5053
        ssl_enabled => true
        ssl_certificate => "/certs/svslogstash01.crt"
        ssl_key => "/certs/private/svslogstash01.key"
        ssl_certificate_authorities => ["/certs/ca_root.crt"]

Here is the auditbeat.yml config for output:

  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: [""]

  ssl_enabled: true
  ssl_certificate_authorities: ["/certs/ca_root.pem"]

Whenever Auditbeat sends logs to Logstash I receive the following errors:

 [2024-05-08T19:48:41,872][WARN ][][linux_auditbeat][linux_auditbeat] An exceptionCaught() event was fired, and it reache
d at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.                                                 
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000002324300000442785ec456cd8edb36179defeba2a

If I disable all of the SSL, then Auditbeat connects and sends logs without issue. The problem is just when enabling SSL.

I have double checked and the svslogstash.crt contains the DNS name and the IP address in the SAN.

On Auditbeat, I have tried using the svslogstash.crt rather than the ca_root.pem.
I have tried setting ssl_client_authentication to none on both Logstash and Auditbeat.

I'm not sure what config I should be using to get them to communicate over SSL.

Have you set ssl_key in the PKCS8 format?

I checked and my key is RSA and it contains with:

Private key stuff
-----End PRIVATE KEY-----

So I believe that it means that it is in PKCS8, correct?

I am still troubleshooting this with no luck thus far.

I verified that the private key matches the crt file by running:

openssl rsa -modulus -in /certs/private/svslogstash01.key -noout | openssl md5
openssl x509 -modulus -in /certs/svslogstash01.crt -noout | openssl md5

I compared the 2 MD5 hashes from those commands and they were identical.

I also installed Apache2 just to check if the certs were working using just straight Apache and SSL. The cert and key worked just fine with Apache.

yum install apache2
ufw allow in 80/tcp
ufw allow in 443/tcp
a2enmod headers
a2enmod rewrite
a2enmod ssl
vim /etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>
  Redirect permanent /
<VirtualHost *:443>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html/
  SSLEngine on
  SSLCertificateFile /certs/svslogstash01.crt
  SSLCertificateKeyFile /certs/private/svslogstash01.key

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

I am running out of ideas and things to try, but I'm still working on it.

Logstash beats input requires a specific key format see here


  • Value type is path
  • There is no default value for this setting.

SSL key to use. This key must be in the PKCS8 format and PEM encoded. You can use the openssl pkcs8 command to complete the conversion. For example, the command to convert a PEM encoded PKCS1 private key to a PEM encoded, non-encrypted PKCS8 key is:

openssl pkcs8 -inform PEM -in path/to/logstash.key -topk8 -nocrypt -outform PEM -out path/to/logstash.pkcs8.key

Running that command against my key, generates a file that is identical to the key.
When I run a md5sum, the two key files are identical according to the hash.
But after changing the permissions on the new pkcs8.key file (550) and updating logstash to point to that new pkcs8.key file, I receive the same Not an SSL/TLS record

To try and clear up any confusion. Here is the command that I used to generate the key and csr originally

openssl req -new -newkey rsa:4096 -nodes –subj "/" -keyout /certs/private/svslogstash01.key -addext ",IP:" -out /certs/svslogstash01.csr

I just read through all of the documentation again, and it appears that all of the certificates should be in PEM format.
I converted the ca_root.crt to PEM, and the svslogstash01.crt to PEM. I did the same thing on the Auditbeat server and updated all of the paths to point to these new PEM certificates.

But I still have the same "Not an SSL/TLS record" whenever I start Auditbeat.


auditbeat test config 

auditbeat test output 
auditbeat test config
Config OK
auditbeat test output
    parse host... OK
    dns lookup... OK
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

Yay! A new error message. but not too many posts about it.

I just checked, and I am running the OSS version of Logstash.

But looking at ELK Subscriptions it shows that Free and open Basic has Secure Settings and Encrypted communications.

Most of the posts that I have found so far say this is caused by using the OSS version. But they are all from a few years ago as well.

It appears that the TLS... WARN secure connection disabled might be a red herring because the talk to server... OK means that the connection is good.

I started to dig through the /etc/logstash/logstash.yml file, which is still set to it's defaults. I found a section that mentioned different TLS algorithms that can be disabled in the file.
I was able to locate the file at /usr/share/logstash/jdk/conf/security/
I edited that file and I commented out the lines that said:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL

Just as a test. I stopped and restarted the logstash service, and started the auditbeat service on another server, and logstash received the same error "Not an SSL/TLS record"

I have already undone that change, since it did not work.

I am just grasping at straws at this point. But I am still hopeful that this can be resolved.

Found it ... Bad syntax..
The above is doing nothing :slight_smile:

  ssl.enabled: true
  ssl.certificate_authorities: ["/certs/ca_root.pem"]

Thank you so much for your help, that solved my problem.

Since there wasn't an explicit definition for Logstash in the documentation, I thought that the syntax was the same between Logstash and Auditbeat.

It's also interesting that the auditbeat config test was successful with a syntax error in the yaml.

But in either case, it is solved now, and I can continue down the path of finishing the configs.

Once again, thank you for all of your help.

1 Like