TLS issue with Filebeat 8.10 an higher

Somecallmesteve · October 24, 2023, 12:03am

I started trying to upgrade some clients to Filebeat 8.10.3 and found that if I install 8.10.x I get the following error when connecting to a Logstash output using tsl :

"...x509: cannot validate certificate for X.X.X.X because it doesn't contain any IP SANs"

Filebeats of version 8.9.X seem to work without an issue though. I looked in the release notes and couldn't see anything that might impact tsl settings. Anyone have any ideas ? Logstash is running at ver 8.10.3.

carly.richmond · October 24, 2023, 10:06am

Hi @Somecallmesteve,

To confirm, are you upgrading from Filebeat 8.9.x to 8.10.x? Do you receive the same issue when upgrading to 8.10.4 (the latest)?

Somecallmesteve · October 29, 2023, 11:04pm

Sorry for the delayed reply @carly.richmond . I just tried 8.10.4 and it has the same issue.

To give more information, I am using the .zip to install and keeping the filebeat.yml from the working 8.9.X install.

stephenb · October 30, 2023, 12:17am

Are you connecting to Elasticsearch by IP or domain name?

Did that change between the versions?

Did you edit the /etc/host file?

Is this a Windows installation?
If so, what version of Windows and have you done? Any updates or upgrades?

Are you using the same certs and CA?

It would also help if you shared your filebeat.yml

Somecallmesteve · October 30, 2023, 12:23am

Filebeat.yml below:

filebeat.inputs:
- type: filestream
  id: iislogstream
  #enabled: false
  paths:
    - C:\inetpub\logs\LogFiles\W3SVC1\*
  exclude_lines: ["^#"]
  tags: ["iislogs"]


  
#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["[X.X.X.X:5045","X.X.X.X:5045","X.X.X.X:5045:5045"]
  ssl.certificate_authorities: ['C:\Program Files\Filebeat\logstash_ca.crt']
  loadbalance: true

stephenb · October 30, 2023, 12:32am

Ok and @Somecallmesteve
Welcome to the community.

Did the version of logstash change?

Also share your input section of log stash where you have the beats input?

Somecallmesteve · October 30, 2023, 12:56am

No it didn't. The only change is the client files with the exception of the filebeat.yml and the logstash_ca.crt .

All I can think of is that something changed from 8.9.X to 8.10.X . I mean it is entirely possible I didn't have something configured properly previously and now it is being enforced.

And the logstash input section:

      client_inactivity_timeout => 10000
      port => 5045
      ssl_enabled => true
      ssl_certificate => "/etc/logstash/certs/logstash_srvchain.crt"
      ssl_key => "/etc/logstash/certs/logstash_srv.key"

stephenb · October 30, 2023, 1:30am

I would look at the SSL settings closely some have been deprecated/ renamed etc

SSL key to use. This key must be in the PKCS8 format and PEM encoded. You can use the openssl pkcs8 command to complete the conversion. For example, the command to convert a PEM encoded PKCS1 private key to a PEM encoded, non-encry

stephenb · October 30, 2023, 1:36am

This is not say there is not a bug or something changed.

There error message implies that there is full certificate validation not just CA.

As you can see here about verification_mode

Filebeat does full validation by default...

Controls the verification of server certificates. Valid values are:

full
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.

.....

So if the IPs are not included in the certificate then the behavior you are seeing now is expected

The default value is full .

Somecallmesteve · October 30, 2023, 1:45am

I guess I will start working my way through some proper troubleshooting and post back if I find anything. I was hoping it would just be something obvious.

Oh well, thanks all for helping rule out something simple I missed.

stephenb · October 30, 2023, 1:47am

In filebeat.yml

Set in the logstash output section

ssl.verification_mode: none

Then try

ssl.verification_mode: certificate

Then try

ssl.verification_mode: full

Silly question did the logstash IPs change?

Somecallmesteve · October 30, 2023, 2:57am

I will give that a go. No it is the same IP. Nothing changes to make the problem occur other than upgrading the client (stop service, overwrite files, start service).

Somecallmesteve · October 30, 2023, 3:05am

So none and certificate work and full fails. This is as I would expect though given it seems to have a problem with the certs SAN's

I guess at least with the certificate option I can upgrade and am still getting encryption over the wire which is the main thing for me.

stephenb · October 30, 2023, 3:30am

Or create new certs with correct IPs

Somecallmesteve · October 30, 2023, 3:31am

The IP's are correct.

stephenb · October 30, 2023, 3:44am

I get the logstash IPs are correct but they do not match or do not exist what is in the certificates that is why the certs fail full validation

You need to create the logstash certs Something like

./bin/elasticsearch-certutil cert \
  --name logstash \
  --ca-cert /path/to/ca/ca.crt \
  --ca-key /path/to/ca/ca.key \
  --dns your.host.name.here \
  --ip 192.0.2.1,192.0.2.2,192.0.2.3 \
  --pem

Here is some instructions this is for agent but same should apply for beats to logstash

You can test the logstash certs with

curl -v --cacert /pathtoca/ca.crt https://logstashIP:9600

Somecallmesteve · October 30, 2023, 3:51am

If I check the logstash cert it has the following lines (the IP's match the logstash input IP's) :

        X509v3 extensions:
            X509v3 Key Usage:
                Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:X.X.X.230, DNS:X.X.X.231, DNS:X.X.X.232, DNS:X.X.X.233

Is this not correct ? And if the certs are not done properly why do they work for the 8.9.X and not the 8.10.X client ?

stephenb · October 30, 2023, 5:12am

To me you have the IPs listed as DNS not IPs is the way I see it..

I Don't know why it worked
..trying to work towards correct now....did you try the curl as I suggested?

It should show you what matches and what does not...

You can also see with example I showed above a cert can contain DNS entries and IP entries...

I am decent with certs it could be something I don't understand... But I do know what works and that is certs with correct DNS and/or IP entries

Somecallmesteve · October 30, 2023, 5:35am

Crud. I see what you mean (IP vs DNS). I think I have been staring at too many log files . I will have a muck around with the cert and post back. The fact that it stopped working from one version to the next is what threw me off the cert scent initially but it really seems like that is the issue.

stephenb · October 30, 2023, 5:35am

Example when I create a cert with this command

./bin/elasticsearch-certutil cert  --name logstash \ 
--ca-cert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt  \ 
--ca-key /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.key \
--dns stephenb-es-8-test \
--ip 10.168.0.12 --pem

Then I run

openssl x509 -text -noout -in certs/logstash/logstash.crt
...

            X509v3 Subject Alternative Name: 
                IP Address:10.168.0.12, DNS:stephenb-es-8-test
            X509v3 Basic Constraints:

And then Beats can do full validation with the IP or DNS but what I see in your certificate is JUST the DNS and you are connecting VIA IPs so the Subject Alternative Names do not match and thus can not do full certificate validation.
That is why you are getting the error... it all lines up.
(Why it did work before I can not answer for you)

So when I run the curl with the IP I see this

$ curl -v --cacert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt https://10.168.0.12:5044
*   Trying 10.168.0.12:5044...
* TCP_NODELAY set
* Connected to 10.168.0.12 (10.168.0.12) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
..........
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=logstash
*  start date: Oct 29 21:02:43 2023 GMT
*  expire date: Oct 28 21:02:43 2026 GMT
*  subjectAltName: host "10.168.0.12" matched cert's IP address!  <!---- HERE 
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.

I even tested when I created a cert with the IP in the DNS SAN ...

$ ./bin/elasticsearch-certutil cert   --name logstashbad   \
--ca-cert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt  
 --ca-key /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.key 
--dns 10.168.0.12 --pem

I get the error

curl -v --cacert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt https://10.168.0.12:5044

* Server certificate:
*  subject: CN=logstashbad
*  start date: Oct 30 05:39:09 2023 GMT
*  expire date: Oct 29 05:39:09 2026 GMT
*  subjectAltName does not match 10.168.0.12
* SSL: no alternative certificate subject name matches target host name '10.168.0.12'

and I see

$ openssl x509 -in logstashbad.crt -noout -text
.....
            X509v3 Subject Alternative Name: 
                DNS:10.168.0.12
            X509v3 Basic Constraints: 
                CA:FALSE

So I just reproduced what you are seeing... (Why it worked before I am unclear) but I am pretty sure that is the problem
You have IPs Address in the DNS SANS Section in you certs and that is why full certificate validation if failing.

I think ...Hmmm why did it change / work before... .. Hmmm

And in general... if I never had to mess with another cert in my career I would be Happy!