Filebeat Cant connect: ERROR x509: cannot validate certificate for x.xx.xx.xxx because it doesn't contain any IP SANs

Peeyush_Gupta · April 6, 2018, 12:05pm

Hi,

I have been trying to connect filebeat to a cluster that has http encryption done, when I try to connect, I get this error:

elasticsearch: https://xx.xx.xx.xx:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xx.xx.xx.xx
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: cannot validate certificate for xx.xx.xx.xx because it doesn't contain any IP SANs

Now, I did a little research and it turned out it might be because by SAN is not set in the certificate, but my certificate has SAN, here is the output:

X509v3 extensions:
            X509v3 Subject Alternative Name:
                IP Address:xx.xx.xx.xx

Any idea what could be causing this issue? TIA.

andrewkroh · April 6, 2018, 4:33pm

Does curl -v --cacert path/to/ca.crt https://xx.xx.xx.xx:9200 work?

Can you use the DNS name associated with the certificate in your filebeat config (output.elasticsearch.hosts: ['https://my.host.com:9200']) instead of the IP?

I would capture the server's certificate chain on the client side and inspect the SANs.

openssl s_client -connect host:9200 -showcerts

Then copy the server cert (it's the first one logged at the top - 0) and run it through openssl. For example:

echo '-----BEGIN CERTIFICATE-----
MIIFJzCCBA+gAwIBAgIQBG0dB5OiJG4lbLsVpw3J2TANBgkqhkiG9w0BAQsFADBN
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMTYwMTE4MDAwMDAwWhcN
MTkwMzI3MTIwMDAwWjBvMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5p
YTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEcMBoGA1UEChMTRWxhc3RpY3NlYXJj
aCwgSW5jLjEVMBMGA1UEAwwMKi5lbGFzdGljLmNvMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA1ZGoE83b/mlYJd55nsMEIreImicqve9I5BfG/TPzj34+
+NzCOiDN4mI6FPP2wd93OHRdyhg4taNZgaraU3F40TSLcWhssm143IgbBJRrfDuU
KWTRFfALl1AVUhsgjotL6IOkXPzbVxonObe0vYDZoUoIwGuPCb9ztvN9ZaTs37ix
P+Hxl83igorrqjVMvos5C6Y4UTucDbS99vqbwCMZfE8cwDOoAR6t2EsEAU7Ha+Ny
AWJO3gFMBWdOHJL4ItszZetkjEbAiFEKhJMYe+BkMjxJfdkAMQ7YvIdfvQYKzrMT
iedT0iBtXG8zJm/KywIdIjXy6zfD+e0raxoH8vQHjQIDAQABo4IB3zCCAdswHwYD
VR0jBBgwFoAUD4BhHIIxYdUvKOeNRji0LOHG2eIwHQYDVR0OBBYEFEp4zo7mkE7Q
Kep0SZcHkRcQ/yXRMCMGA1UdEQQcMBqCDCouZWxhc3RpYy5jb4IKZWxhc3RpYy5j
bzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MGsGA1UdHwRkMGIwL6AtoCuGKWh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9zc2Nh
LXNoYTItZzUuY3JsMC+gLaArhilodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vc3Nj
YS1zaGEyLWc1LmNybDBMBgNVHSAERTBDMDcGCWCGSAGG/WwBATAqMCgGCCsGAQUF
BwIBFhxodHRwczovL3d3dy5kaWdpY2VydC5jb20vQ1BTMAgGBmeBDAECAjB8Bggr
BgEFBQcBAQRwMG4wJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNv
bTBGBggrBgEFBQcwAoY6aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0U0hBMlNlY3VyZVNlcnZlckNBLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3
DQEBCwUAA4IBAQBAYIt6lwPB5WANbDV58qeskp+QAcLI3F6ePf8Go4COoGmCKvoD
L11F4/x3TbrU4984f2aFoa499CjE+LNh3MEWe1T1cBWstAi0VlsLmO/cDY0gPENK
bON53K4pUEntcSbF/ptQ6Z0aDotBOdv4Wl0pCE4r4kneWwB5yfpfM5iobIZvQFw1
2Fu8+3ILzL2XJeB9Xn2/YF6vr9KFFGzzTCSlkk2HH32CerBkzgLT9w05Tb+cxdCP
i9yxKfd0yA7IE42dYBRB25ir7OnxvyJfzk8wCnlT6V2tnzXqBe/Ql3XpJHeOFyzH
vqRI9Xt2dnAvH5uv6cGv9zs29rpm+bBdESdL
-----END CERTIFICATE-----' | openssl x509 -noout -text

Peeyush_Gupta · April 7, 2018, 6:08am

I tried to capture using openssl s_client -connect host:9200 -showcerts, and the certificate it gave has the subject name as: "Elastic Certificate Tool Autogenerated CA", now if I use that certificate to connect, I get an error:

curl: (51) SSL: certificate subject name (Elastic Certificate Tool Autogenerated CA) does not match target host name 'host'

Not sure, how to change the subject name to make it connect.

Sorry for the newbie level questions, I am new to ssl.

andrewkroh · April 9, 2018, 2:14am

It sounds like there's an issue on the server side. Did you use the certutil to generate the CA cert and the ES server cert?

That sounds like a CA certificate and not a server certificate.

Peeyush_Gupta · April 9, 2018, 10:51am

I did use cerutil and after making some config related changes, here is what I am getting now:

* Rebuilt URL to: https://instance:9200/
*   Trying xx.xx.xx.xx...
* Connected to instance (xx.xx.xx.xx) port 9200 (#0)
* found 1 certificates in ca.crt
* found 598 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_CBC_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: instance (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=instance
* 	 start date: Mon, 09 Apr 2018 09:01:34 GMT
* 	 expire date: Thu, 08 Apr 2021 09:01:34 GMT
* 	 issuer: CN=Elastic Certificate Tool Autogenerated CA
* 	 compression: NULL
* ALPN, server did not agree to a protocol
> GET / HTTP/1.1
> Host: instance:9200
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 369
<
* Connection #0 to host instance left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Looks like some issues with authorization, any idea how to debug it?

Peeyush_Gupta · April 9, 2018, 11:24am

So curl based authentication works:

* Rebuilt URL to: https://instance:9200/
*   Trying xx.xx.xx.xx...
* Connected to instance (xx.xx.xx.xx) port 9200 (#0)
* found 152 certificates in /etc/ssl/certs/ca-certificates.crt
* found 599 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_CBC_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: instance (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=instance
* 	 start date: Mon, 09 Apr 2018 09:01:34 GMT
* 	 expire date: Thu, 08 Apr 2021 09:01:34 GMT
* 	 issuer: CN=Elastic Certificate Tool Autogenerated CA
* 	 compression: NULL
* ALPN, server did not agree to a protocol
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: instance:9200
> Authorization: Basic ZWxhc3RpYzprY1lSMUZyNUNmcFBxT2xsMG1oTw==
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 435
<
{
  "name" : "35kkdFc",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "fQNBalJJRIqetaVo73X4LQ",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "c59ff00",
    "build_date" : "2018-03-13T10:06:29.741383Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host instance left intact

Still not sure, how to use this in filebeat.yml.
As I understand, I can capture the crt, but where to get the key from?

andrewkroh · April 9, 2018, 7:12pm

So if you have curl working it should just be matter of setting the equivalent settings on your filebeat.yml. I don't know exactly what arguments you used with curl so I'm guessing that you set a CA cert and a basic auth user/pass.

output.elasticsearch.hosts: ['https://instance:9200']
output.elasticsearch.username: myuser
output.elasticsearch.password: secret
output.elasticsearch.ssl.certificate_authorities: ['/path/to/ca.crt']

Peeyush_Gupta · April 10, 2018, 6:10am

I am using this command for curl:

curl -v https://instance:9200 --user elastic:kcYR1Fr5CfpP0mhO --cacert ca.crt

I have updated the filebeat.yml with the appropriate values but
I am still getting that cannot validate error while using filebeat

andrewkroh · April 10, 2018, 2:53pm

Then the config options I gave should be correct for Filebeat. Can you please post your latest configuration and the output from filebeat test output -e -d "*".

Peeyush_Gupta · April 12, 2018, 6:39am

Here is the output:

./filebeat test output -e -d "*"
2018-04-12T02:35:19.850-0400	INFO	instance/beat.go:479	Home path: [/root/go/src/github.com/elastic/beats/filebeat] Config path: [/root/go/src/github.com/elastic/beats/filebeat] Data path: [/root/go/src/github.com/elastic/beats/filebeat/data] Logs path: [/root/go/src/github.com/elastic/beats/filebeat/logs]
2018-04-12T02:35:19.850-0400	DEBUG	[beat]	instance/beat.go:506	Beat metadata path: /root/go/src/github.com/elastic/beats/filebeat/data/meta.json
2018-04-12T02:35:19.850-0400	INFO	instance/beat.go:486	Beat UUID: e290a366-a0bc-4ba4-8a9a-2039f433aa69
2018-04-12T02:35:19.851-0400	INFO	elasticsearch/client.go:145	Elasticsearch url: https://xx.xx.xx.xx:9200
elasticsearch: https://xx.xx.xx.xx:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xx.xx.xx.xx
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: cannot validate certificate for xx.xx.xx.xx because it doesn't contain any IP SANs

Here is my filebeat config:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xx.xx.xx.xx:9200"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "kcYR1Fr5Cll0mhO"
  ssl.certificate_authorities: "/path / to /ca/crt"

Peeyush_Gupta · April 12, 2018, 8:03am

Figured it out. I was using IP instead of hostname in filebeat.yml.

Thanks.

andrewkroh · April 12, 2018, 8:21am

Excellent. I was running out of ideas.

system · May 10, 2018, 8:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.