Kibana to Elastic Search Communication Not Working with Wildcard Certificate

For the last couple of days I have been trying to add a wildcard certificate to a new ELK server (one server running ES+Kibana). I have gotten the cert to work with ES, however, Kibana is unable to communicate with ES using certain settings. Hopefully I'm missing something simple. Here is my setup...

OpenSSL (possibly overkill)

// ORIG.pfx is a Sectigo RSA wildcard, non-root certificate
openssl.exe pkcs12 -in "ORIG.pfx" -clcerts -nokeys -out "newes-crt.pem"
openssl.exe pkcs12 -in "ORIG.pfx" -out "KEY.pem" -nodes
openssl.exe rsa -in "KEY.pem" -out "KEYRSA.key"
openssl.exe pkcs8 -in "KEYRSA.key" -topk8 -nocrypt -out "newes-key.pem"

Working

// elasticsearch.yml
cluster.name: elk-dev
node.name: elk-elastic
path.data: E:\\ELK\\es-data
path.logs: E:\\ELK\\es-logs
network.host: 10.10.10.13
discovery.type: single-node
http.port: 9200
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: optional
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem

// kibana.yml
server.port: 5601
server.host: 10.10.10.13
server.publicBaseUrl: "https://host.domain.com:5601"
server.name: "elk-kibana"
elasticsearch.hosts: ["https://host.domain.com:9200"]
elasticsearch.serviceAccountToken: "<token>"
server.ssl.enabled: true
server.ssl.certificate: E:\ELK\es\config\certs\newes-crt.pem
server.ssl.key: E:\ELK\es\config\certs\newes-key.pem
elasticsearch.ssl.verificationMode: none
logging.root.level: all

No errors but I'm assuming that TLS is not working between them, though they are on the same machine. Not sure if this is an appropriate set up. Guessing this config would break if the servers were separated.

Not Working 1

// elasticsearch.yml
cluster.name: elk-dev
node.name: elk-elastic
path.data: E:\\ELK\\es-data
path.logs: E:\\ELK\\es-logs
network.host: 10.10.10.13
discovery.type: single-node
http.port: 9200
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: optional
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem

// kibana.yml
server.port: 5601
server.host: 10.10.10.13
server.publicBaseUrl: "https://host.domain.com:5601"
server.name: "elk-kibana"
elasticsearch.hosts: ["https://host.domain.com:9200"]
elasticsearch.username: "kibana-system"
elasticsearch.password: "<password>"
server.ssl.enabled: true
server.ssl.certificate: E:\ELK\es\config\certs\newes-crt.pem
server.ssl.key: E:\ELK\es\config\certs\newes-key.pem
elasticsearch.ssl.verificationMode: none
logging.root.level: all

[security_exception]: unable to authenticate user [kibana-system] for REST request

Not Working 2

// elasticsearch.yml
cluster.name: elk-dev
node.name: elk-elastic
path.data: E:\\ELK\\es-data
path.logs: E:\\ELK\\es-logs
network.host: 10.10.10.13
discovery.type: single-node
http.port: 9200
xpack.security.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem
xpack.security.transport.ssl:
  enabled: true
  verification_mode: full
  client_authentication: required
  key: certs/newes-key.pem
  certificate: certs/newes-crt.pem

// kibana.yml
server.port: 5601
server.host: 10.10.10.13
server.publicBaseUrl: "https://host.domain.com:5601"
server.name: "elk-kibana"
elasticsearch.hosts: ["https://host.domain.com:9200"]
elasticsearch.username: "kibana-system"
elasticsearch.password: "<password>"
server.ssl.enabled: true
server.ssl.certificate: E:\ELK\es\config\certs\newes-crt.pem
server.ssl.key: E:\ELK\es\config\certs\newes-key.pem
elasticsearch.ssl.verificationMode: full
logging.root.level: all

[ConnectionError]: unable to verify the first certificate

The only changes between the different configs are "verification_mode" and "client_authentication". "Not Working 2" is what I think is the proper configuration, as in more correct than the "Working" config. At least in terms of adding HTTPS/TLS security. Maybe I am wrong in that and would appreciate being corrected. I should say that I don't know much about ELK.

I should also say that I found the following thread but sadly it didn't help, though it seemed like the exact same issue. It returned the same "unable to verify the first certificate" error.

Thank you all for your help and have a nice weekend!

So for "Not working 1" the only difference I see is the token vs user/pass authentication. Am i missing something there?
And if you were trying to use the built in Kibana user, the name is kibana_system instead of kibana-system so that's why you're getting the authentication error.

As for the "Not working 2" it fails because the client_authentication setting is used for PKI authentication which doesn't seem to be something that you're trying to setup here:

Guess I'm a little bit confused. I was referencing this page and it doesn't say anything about PKI. Possibly because it is a general breakdown of settings. Anyway, it simply shows "client_authentication" and "verification_mode" under "xpack.security.transport.ssl", which seemed like what I am looking for.

The ES username and password is working now. Thanks for that. Not sure about that typo, thought I copied that from the initial setup.

Anyway, I still cannot get Kibana to connect with anything other than "elasticsearch.ssl.verificationMode: none". ES has "verification_mode" set to "certificate and "client_authentication" set to "optional" (courtesy of this Github page). Currently, ES and Kibana are on the same server. Do I not need "verificationMode set to "certificate" or "full" because of that? I'd like to have it on in case we move them to separate servers later.

As it stands, HTTPS is valid for: browser to ES and browser to Kibana. But receiving "[ConnectionError]: unable to verify the first certificate" for what I assume is Kibana to ES. Is this not possible without PKI?

Hello. Still curious if anyone has any more input on this. Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.