Auditbeat to Logstash SSL errors with "Not an SSL/TLS record"

Elastic Stack Logstash
1

My goal is connect Auditbeat to Logstash and encrypt the communication. I am not worried about having Logstash verify the client. I do have client certificates on some servers, but I would prefer not to use them, as I don't want to create client certificates on all servers.

I am running my own CA, with self generated certificates.

Logstash version 8.11.0
Auditbeat version 8.11.1

Here is my Logstash input for Auditbeat:

input {
    beats {
        id => "linux_auditbeat"
        port => 5053
        ssl_enabled => true
        ssl_certificate => "/certs/svslogstash01.crt"
        ssl_key => "/certs/private/svslogstash01.key"
        ssl_certificate_authorities => ["/certs/ca_root.crt"]
    }
}

Here is the auditbeat.yml config for output:

output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: ["svslogstash01.corp.survios.com:5053"]

  ssl_enabled: true
  ssl_certificate_authorities: ["/certs/ca_root.pem"]

Whenever Auditbeat sends logs to Logstash I receive the following errors:

 [2024-05-08T19:48:41,872][WARN ][io.netty.channel.DefaultChannelPipeline][linux_auditbeat][linux_auditbeat] An exceptionCaught() event was fired, and it reache
d at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.                                                 
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000002324300000442785ec456cd8edb36179defeba2a
f619c550b501aead73657c9268ba2bba440d14130a0c82b9bb524aa2435893bf05bf7010ad0f24f32f919a00132abb148de7b79ce3de732ffe5e6e6e67f37373ffcfe8817c1f4e483ec4708e43c2f13
5e257cf5265b8b72258a755ad4f51f6078d153905a0609f188866480809cb409f17f86b01f0902f7da2a303c90f3c60e1058a559966638306cad0f108f507608d20ce4ccdfa4215ad979623023c41d3
29e66699e66590e8696565c88bce2a2aadb56b4941742b699c45b865e2a883b709e543ca9ea242f9297af92ec25de1e130db22708f807ef4722cdb3545937a67e720fc6fa54d91e0cd2a9ad09a4c2e4
0802ef57f57d5d82e15987ad8778442b7bd3ed21a0a93172b81cfead9986308161476ea00e02555aa63cc996cb64430339a3c0a0aca6395b6b95ec2e40766698de8361ec6468adeb21309d425ed0cd7
9cacbb45efcfae6f5e2a75756c96ef1ca7a2f7fc681c16808146ab92e56759d978d2ef355d52cf3a25c565ae5755635cb020706b9a12152339378cdeb25d7cc24038d5beac9c9ee3e6628d7956acbba
4c3229eba4a48c12c9eb2cc9979aeab259570dad305723d7ebba2cda555235cb26292bce93262fd749d3e6bcacda82675a3f8f8003033dcc65f7564f1d9d1a52e3c0e0c93d18451f5f2aae9173d6413
ca227efe58620f07a67c6d10c9b859b3a5a283bb4663339198c1dc4e2e5a44d882b7e211d2d3aab76748ca462135c63c4538ec3212aecff37373ffef32c8515df4661a3b38a7c2c69341a62c58b9aaf
1946e966a44e9fab757560082644d86e27ef6efd563aba3d537fdb98e1f26b9124a30cdb746b7b5a7c6afb69c311bac52d05f5f4b496412e6e1fa4bbed4c7359be107ece0e067a4f6a0ab27966855f2
3fde43edfc36766857f79d3e7cce8e83317ba3f2f79f63ddce8d93ec33e69fcd7ee13dde4db39c479265d77d505c7ff64675fd0fe952f49158e4b6a2b870de92456927c602f60b05350f63871261505
ccb0338386982dee7493bb39d011cb818ee119ccd0da38143ff2410625036dacdb43dc9ddd211271f137300c14de59b7c3db03c3e4297ae3dc07ceda00062f1f483ffdbc71761acfd4f18b90e3b1536
bf03863266de28c892c67bcca6a9e65fc6adcee6848a5eecdf0e2c9a83e30b4a623bff781fa0ff27db582eb7b5c0afaeac1c30515f1086ddc0cb6006d1c797fbe5244c5d35f130d8a20caa228ca9cc1
919fbaf001a5a7c7d379a995a68b466702046a8e39e43d0db2e94843200383ed3404f2b8fa143cbff74a761d043c0d3a5830c862be91270f812ccb4b30d831ee096090190414e77c9535858a7939049
66008610f8131f80c0c328f0f083078ab7614e93b3f78060a9d1976b139b476b1ae4fff45ac9cda5e9e587140fbf98538d7e7a7be976e3febe638a047678edfbed41c0c9e941d743c3cd3c7609b3f49
c58a8fda3936e0ac3db0abd09e62676cedbb670f9af9e1703f873e0afa5e6de5b0211c0eff0e00542a86b7

If I disable all of the SSL, then Auditbeat connects and sends logs without issue. The problem is just when enabling SSL.

I have double checked and the svslogstash.crt contains the DNS name and the IP address in the SAN.

On Auditbeat, I have tried using the svslogstash.crt rather than the ca_root.pem.
I have tried setting ssl_client_authentication to none on both Logstash and Auditbeat.

I'm not sure what config I should be using to get them to communicate over SSL.

2

Have you set ssl_key in the PKCS8 format?

3

I checked and my key is RSA and it contains with:

-----BEGIN PRIVATE KEY-----
Private key stuff
-----End PRIVATE KEY-----

So I believe that it means that it is in PKCS8, correct?

4

I am still troubleshooting this with no luck thus far.

I verified that the private key matches the crt file by running:

openssl rsa -modulus -in /certs/private/svslogstash01.key -noout | openssl md5
openssl x509 -modulus -in /certs/svslogstash01.crt -noout | openssl md5

I compared the 2 MD5 hashes from those commands and they were identical.

I also installed Apache2 just to check if the certs were working using just straight Apache and SSL. The cert and key worked just fine with Apache.

yum install apache2
ufw allow in 80/tcp
ufw allow in 443/tcp
a2enmod headers
a2enmod rewrite
a2enmod ssl
vim /etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>
  ServerName svslogstash01.corp.survios.com
  Redirect permanent / https://svslogstash01.corp.survios.com/
</VirtualHost>
<VirtualHost *:443>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html/
  ServerName svslogstash01.corp.survios.com
  SSLEngine on
  SSLCertificateFile /certs/svslogstash01.crt
  SSLCertificateKeyFile /certs/private/svslogstash01.key

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

I am running out of ideas and things to try, but I'm still working on it.

5

Logstash beats input requires a specific key format see here

ssl_key

  • Value type is path
  • There is no default value for this setting.

SSL key to use. This key must be in the PKCS8 format and PEM encoded. You can use the openssl pkcs8 command to complete the conversion. For example, the command to convert a PEM encoded PKCS1 private key to a PEM encoded, non-encrypted PKCS8 key is:

openssl pkcs8 -inform PEM -in path/to/logstash.key -topk8 -nocrypt -outform PEM -out path/to/logstash.pkcs8.key

6

Running that command against my key, generates a file that is identical to the key.
When I run a md5sum, the two key files are identical according to the hash.
But after changing the permissions on the new pkcs8.key file (550) and updating logstash to point to that new pkcs8.key file, I receive the same Not an SSL/TLS record

7

To try and clear up any confusion. Here is the command that I used to generate the key and csr originally

openssl req -new -newkey rsa:4096 -nodes –subj "/emailAddress=it@domain.com/countryName=US/organizationName=Domain/commonName=svslogstash01.corp.survios.com" -keyout /certs/private/svslogstash01.key -addext "subjectAltName=DNS:svslogstash01.corp.survios.com,IP:10.1.2.121" -out /certs/svslogstash01.csr
8

I just read through all of the documentation again, and it appears that all of the certificates should be in PEM format.
I converted the ca_root.crt to PEM, and the svslogstash01.crt to PEM. I did the same thing on the Auditbeat server and updated all of the paths to point to these new PEM certificates.

But I still have the same "Not an SSL/TLS record" whenever I start Auditbeat.

9

Run

auditbeat test config 

auditbeat test output
10 
auditbeat test config
Config OK

auditbeat test output
logstash: svslogstash01.corp.survios.com:5053...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.1.2.121
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

Yay! A new error message. but not too many posts about it.

11

I just checked, and I am running the OSS version of Logstash.

But looking at ELK Subscriptions it shows that Free and open Basic has Secure Settings and Encrypted communications.

Most of the posts that I have found so far say this is caused by using the OSS version. But they are all from a few years ago as well.

12

It appears that the TLS... WARN secure connection disabled might be a red herring because the talk to server... OK means that the connection is good.

I started to dig through the /etc/logstash/logstash.yml file, which is still set to it's defaults. I found a section that mentioned different TLS algorithms that can be disabled in the java.security file.
I was able to locate the java.security file at /usr/share/logstash/jdk/conf/security/java.security.
I edited that file and I commented out the lines that said:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL

Just as a test. I stopped and restarted the logstash service, and started the auditbeat service on another server, and logstash received the same error "Not an SSL/TLS record"

I have already undone that change, since it did not work.

I am just grasping at straws at this point. But I am still hopeful that this can be resolved.

13

Found it ... Bad syntax..
The above is doing nothing :slight_smile:

  ssl.enabled: true
  ssl.certificate_authorities: ["/certs/ca_root.pem"]
14

Thank you so much for your help, that solved my problem.

Since there wasn't an explicit definition for Logstash in the documentation, I thought that the syntax was the same between Logstash and Auditbeat.

It's also interesting that the auditbeat config test was successful with a syntax error in the yaml.

But in either case, it is solved now, and I can continue down the path of finishing the configs.

Once again, thank you for all of your help.

1 Like