TLS issue with Filebeat 8.10 an higher

Elastic Stack Beats
1

I started trying to upgrade some clients to Filebeat 8.10.3 and found that if I install 8.10.x I get the following error when connecting to a Logstash output using tsl :

"...x509: cannot validate certificate for X.X.X.X because it doesn't contain any IP SANs"

Filebeats of version 8.9.X seem to work without an issue though. I looked in the release notes and couldn't see anything that might impact tsl settings. Anyone have any ideas ? Logstash is running at ver 8.10.3.

2

Hi @Somecallmesteve,

To confirm, are you upgrading from Filebeat 8.9.x to 8.10.x? Do you receive the same issue when upgrading to 8.10.4 (the latest)?

3

Sorry for the delayed reply @carly.richmond . I just tried 8.10.4 and it has the same issue.

To give more information, I am using the .zip to install and keeping the filebeat.yml from the working 8.9.X install.

4

Are you connecting to Elasticsearch by IP or domain name?

Did that change between the versions?

Did you edit the /etc/host file?

Is this a Windows installation?
If so, what version of Windows and have you done? Any updates or upgrades?

Are you using the same certs and CA?

It would also help if you shared your filebeat.yml

5

Filebeat.yml below:

filebeat.inputs:
- type: filestream
  id: iislogstream
  #enabled: false
  paths:
    - C:\inetpub\logs\LogFiles\W3SVC1\*
  exclude_lines: ["^#"]
  tags: ["iislogs"]


  
#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["[X.X.X.X:5045","X.X.X.X:5045","X.X.X.X:5045:5045"]
  ssl.certificate_authorities: ['C:\Program Files\Filebeat\logstash_ca.crt']
  loadbalance: true
6

Ok and @Somecallmesteve
Welcome to the community.

Did the version of logstash change?

Also share your input section of log stash where you have the beats input?

7

No it didn't. The only change is the client files with the exception of the filebeat.yml and the logstash_ca.crt .

All I can think of is that something changed from 8.9.X to 8.10.X . I mean it is entirely possible I didn't have something configured properly previously and now it is being enforced.

And the logstash input section:

      client_inactivity_timeout => 10000
      port => 5045
      ssl_enabled => true
      ssl_certificate => "/etc/logstash/certs/logstash_srvchain.crt"
      ssl_key => "/etc/logstash/certs/logstash_srv.key"
8

I would look at the SSL settings closely some have been deprecated/ renamed etc

SSL key to use. This key must be in the PKCS8 format and PEM encoded. You can use the openssl pkcs8 command to complete the conversion. For example, the command to convert a PEM encoded PKCS1 private key to a PEM encoded, non-encry

9

This is not say there is not a bug or something changed.

There error message implies that there is full certificate validation not just CA.

As you can see here about verification_mode

Filebeat does full validation by default...

Controls the verification of server certificates. Valid values are:

full
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.

.....

So if the IPs are not included in the certificate then the behavior you are seeing now is expected

The default value is full .

10

I guess I will start working my way through some proper troubleshooting and post back if I find anything. I was hoping it would just be something obvious.

Oh well, thanks all for helping rule out something simple I missed.

11

In filebeat.yml

Set in the logstash output section

ssl.verification_mode: none

Then try

ssl.verification_mode: certificate

Then try

ssl.verification_mode: full

Silly question did the logstash IPs change?

12

I will give that a go. No it is the same IP. Nothing changes to make the problem occur other than upgrading the client (stop service, overwrite files, start service).

13

So none and certificate work and full fails. This is as I would expect though given it seems to have a problem with the certs SAN's

I guess at least with the certificate option I can upgrade and am still getting encryption over the wire which is the main thing for me.

14

Or create new certs with correct IPs

15

The IP's are correct.

16

I get the logstash IPs are correct but they do not match or do not exist what is in the certificates that is why the certs fail full validation

You need to create the logstash certs Something like

./bin/elasticsearch-certutil cert \
  --name logstash \
  --ca-cert /path/to/ca/ca.crt \
  --ca-key /path/to/ca/ca.key \
  --dns your.host.name.here \
  --ip 192.0.2.1,192.0.2.2,192.0.2.3 \
  --pem

Here is some instructions this is for agent but same should apply for beats to logstash

You can test the logstash certs with

curl -v --cacert /pathtoca/ca.crt https://logstashIP:9600

17

If I check the logstash cert it has the following lines (the IP's match the logstash input IP's) :

        X509v3 extensions:
            X509v3 Key Usage:
                Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:X.X.X.230, DNS:X.X.X.231, DNS:X.X.X.232, DNS:X.X.X.233

Is this not correct ? And if the certs are not done properly why do they work for the 8.9.X and not the 8.10.X client ?

18

To me you have the IPs listed as DNS not IPs is the way I see it..

I Don't know why it worked
..trying to work towards correct now....did you try the curl as I suggested?

It should show you what matches and what does not...

You can also see with example I showed above a cert can contain DNS entries and IP entries...

I am decent with certs it could be something I don't understand... But I do know what works and that is certs with correct DNS and/or IP entries

19

Crud. I see what you mean (IP vs DNS). I think I have been staring at too many log files :dizzy_face:. I will have a muck around with the cert and post back. The fact that it stopped working from one version to the next is what threw me off the cert scent initially but it really seems like that is the issue.

1 Like
20

Example when I create a cert with this command

./bin/elasticsearch-certutil cert  --name logstash \ 
--ca-cert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt  \ 
--ca-key /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.key \
--dns stephenb-es-8-test \
--ip 10.168.0.12 --pem

Then I run

openssl x509 -text -noout -in certs/logstash/logstash.crt
...

            X509v3 Subject Alternative Name: 
                IP Address:10.168.0.12, DNS:stephenb-es-8-test
            X509v3 Basic Constraints:

And then Beats can do full validation with the IP or DNS but what I see in your certificate is JUST the DNS and you are connecting VIA IPs so the Subject Alternative Names do not match and thus can not do full certificate validation.
That is why you are getting the error... it all lines up.
(Why it did work before I can not answer for you)

So when I run the curl with the IP I see this

$ curl -v --cacert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt https://10.168.0.12:5044
*   Trying 10.168.0.12:5044...
* TCP_NODELAY set
* Connected to 10.168.0.12 (10.168.0.12) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
..........
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=logstash
*  start date: Oct 29 21:02:43 2023 GMT
*  expire date: Oct 28 21:02:43 2026 GMT
*  subjectAltName: host "10.168.0.12" matched cert's IP address!  <!---- HERE 
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.

I even tested when I created a cert with the IP in the DNS SAN ...

$ ./bin/elasticsearch-certutil cert   --name logstashbad   \
--ca-cert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt  
 --ca-key /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.key 
--dns 10.168.0.12 --pem

I get the error

curl -v --cacert /home/sbrown/logstash/elasticsearch-8.10.4/certs/ca/ca.crt https://10.168.0.12:5044

* Server certificate:
*  subject: CN=logstashbad
*  start date: Oct 30 05:39:09 2023 GMT
*  expire date: Oct 29 05:39:09 2026 GMT
*  subjectAltName does not match 10.168.0.12
* SSL: no alternative certificate subject name matches target host name '10.168.0.12'

and I see

$ openssl x509 -in logstashbad.crt -noout -text
.....
            X509v3 Subject Alternative Name: 
                DNS:10.168.0.12
            X509v3 Basic Constraints: 
                CA:FALSE

So I just reproduced what you are seeing... (Why it worked before I am unclear) but I am pretty sure that is the problem
You have IPs Address in the DNS SANS Section in you certs and that is why full certificate validation if failing.

I think ...Hmmm why did it change / work before... .. Hmmm

And in general... if I never had to mess with another cert in my career I would be Happy!

1 Like