I can't get Elastic and Kibana to work with real certificates

Elastic Stack Elasticsearch
1

I can't get Elastic and Kibana to work with real certificates from either Let's encrypt or Comodo.

I've thought about it a thousand times, but no matter how much I read and reread, I can't find the error of something as simple as installing certificate chains. At first I thought it was a Let's Encrypt issue but this afternoon I bought some certificates issued by Comodo and nothing, the same.

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
17:21:59.701 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [91.121.226.53]; the server provided a certificate with subject name [CN=elk2.mydomain.ovh], fingerprint [f611c9e9ce2462ed1f3c7e29035f2db2347e39df], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:elk2.mydomain.ovh,DNS:www.elk2.mydomain.ovh]; the certificate is issued by [CN=Don Dominio / MrDomain RSA DV CA,O=Soluciones Corporativas IP\, SL,L=Manacor,ST=Illes Balears,C=ES] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [ce1710a80bd0dfb6ed688c37c8fc066268bd814b] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/elk2.mydomain.ovh.ca.crt})])

/etc/kibana/kibana.yml

server.host: "91.121.226.53"
server.publicBaseUrl: "https://elk2.mydomain.ovh:5601"
server.name: "elk2.mydomain.ovh"
server.ssl.enabled: true
server.ssl.certificate: certs/elk2.mydomain.ovh.crt
server.ssl.key: certs/elk2.mydomain.ovh.key
server.ssl.certificateAuthorities: ["certs/elk2.mydomain.ovh.ca.crt"]
elasticsearch.hosts: ["https://elk2.mydomain.ovh:9200"]
elasticsearch.ssl.certificate:  certs/elk2.mydomain.ovh.crt
elasticsearch.ssl.key: certs/elk2.mydomain.ovh.key
elasticsearch.ssl.certificateAuthorities: [ "certs/elk2.mydomain.ovh.ca.crt" ]
elasticsearch.ssl.verificationMode:  none
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

Elasticsearch

root@elk2:/etc/ssl# grep -Ev '^#|^$'  /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: certs/elk2.mydomain.ovh.key
  certificate: certs/elk2.mydomain.ovh.crt
  certificate_authorities: ["certs/elk2.mydomain.ovh.ca.crt"]
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: certs/elk2.mydomain.ovh.key
  certificate: certs/elk2.mydomain.ovh.crt
  certificate_authorities: ["certs/elk2.mydomain.ovh.ca.crt"]
cluster.initial_master_nodes: ["elk2.mydomain.ovh"]
http.host: 0.0.0.0

Of course I have verified the files found in both the elasticsearch and kibana certs, just as I do with any software installation that requires certificates, including authority or CA

Apreciate help.

2

Hi @abkrim

Pretty sure that's the transport layer that is having an issue.

I believe Transport tends to presents the IP address this 91.121.226.53 which is not in the certificate that you provided.

So set that verification mode to none or for the transport SSL or just use the self-sign certs and they're generated using the IPs of the hosts they are on.

By the way, I use lets encrypt all the time and I just use the self sign for the transport layer.

You could try this but don't know if that will work...

network.host: elk2.mydomain.ovh

3

I've changed on elasticsearch.yml network.host: elk2. mydomain.ovh

Try restarting and the same problem.

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: elk2.mydomain.ovh
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: certs/privkey.pem
  certificate: certs/fullchain.pem
  certificate_authorities: [
    "certs/chain.pem",
    "certs/cacert.x1.pem"
  ]
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: certs/privkey.pem
  certificate: certs/fullchain.pem
  certificate_authorities: [
    "certs/chain.pem",
    "certs/cacert.x1.pem"
  ]
cluster.initial_master_nodes: ["elk2.mydomain.ovh"]
http.host: 0.0.0.0

Error

18:42:54.501 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [91.121.226.53]; the server provided a certificate with subject name [CN=elk2.endesarrollo.ovh], fingerprint [d3432f5a03043e533bd1a7a1aeabce8964bd1ac2], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:elk2.endesarrollo.ovh]; the certificate is issued by [CN=R3,O=Let's Encrypt,C=US]; the certificate is signed by (subject [CN=R3,O=Let's Encrypt,C=US] fingerprint [a053375bfe84e8b748782c7cee15827a6af5a405] {trusted issuer}) signed by (subject [CN=ISRG Root X1,O=Internet Security Research Group,C=US] fingerprint [933c6ddee95c9c41a40f9f50493d82be03ad87bf] {trusted issuer}) which is issued by [CN=DST Root CA X3,O=Digital Signature Trust Co.] (but that issuer certificate was not provided in the chain); this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/chain.pem,/etc/elasticsearch/certs/cacert.x1.pem})]) is not configured to trust that issuer but trusts [2] other issuers ([CN=ISRG Root X1,O=Internet Security Research Group,C=US, CN=R3,O=Let's Encrypt,C=US])
java.security.cert.CertificateException: No subject alternative names matching IP address 91.121.226.53 found
	at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165) ~[?:?]
4

Yeah I wasn't sure that was going to work. That's why I told you to use the self-sign cert or verification to none for the transport.

You can see the transport is providing the IP name and your certificate only has a hostname. It will never match.

Or you're going to have to add that IP to your certificate or use a different cert.

Here is my fully working elasticsearch.yml the http endpoint uses a Let's Encrypt cert for http and the defaults self signed for the transport layer.

root@stephenb-es-8-test:~# cat /etc/elasticsearch/elasticsearch.yml  | grep "^[^#;]"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  certificate: certs/fullchain.pem 
  key: certs/privkey.pem
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["stephenb-es-8-test"]
http.host: 0.0.0.0

Actually reset password should not use the transport layer. Let me look at this little bit more.

5

Hi @abkrim Apologies Disregard most of that :slight_smile: (embarassing!!)

You just need to provide your FQDN to reset password command (sorry for the digression... I was thinking about the transport layer)

This should work
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -url https://elk2.mydomain.ovh:9200

since you don't provide a url it just uses the IP address...which then does not match!

I just tested on mine I got the same error then put in the FQDN and it worked fine!

1 Like
6

Oh my god... a lot of thanks.

Don't worry, I also got stubborn reading the documentation, and I didn't see the need to use --url because the message is very clear in the installer.

A lack of usability in the manual or better, in the message that should be sent to the manual corrected.

Thank you very much.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.