Setup Elasticsearch and Kibana on the same server with TLS/SSL

Elastic Stack Elasticsearch
1

I have tried for the past week to get ES and Kibana working over https on the same server with proper DNS names. :confused:

I am using a wildcard certificate for everything e.g. ES DNS, Kibana DNS and server certificate.

I have succesfully setup ES with https and I am able to connect to it from outside the server with my DNS name specified in the certificate. The problem occurs when I am trying to run Kibana.. it seems it does not want to connect to ES over https (even though it is on the same server).

Can anyone see what I am doing wrong?

Elasticsearch.yml

bootstrap.memory_lock: true
network.host: 0.0.0.0
node.name: node-syst
path.data: F:\Elasticsearch\Data
path.logs: F:\Elasticsearch\Logs

xpack.license.self_generated.type: basic
xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/wildcard_cert.key
xpack.security.http.ssl.certificate: certs/wildcard_cert.crt
xpack.security.http.ssl.certificate_authorities: certs/wildcard_cert.ca.crt

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/wildcard_cert.key
xpack.security.transport.ssl.certificate: certs/wildcard_cert.crt
xpack.security.transport.ssl.certificate_authorities: certs/wildcard_cert.ca.crt

discovery.seed_hosts: node-syst.sigmaestimates.com

Kibana.yml

server.name: kibana-syst
server.ssl.enabled: true
server.ssl.key: certs/wildcard_cert.key
server.ssl.certificate: certs/wildcard_cert.crt
xpack_security_enabled: true
elasticsearch.hosts: https://node-syst.sigmaestimates.com:9200
elasticsearch.username: "kibana"
elasticsearch.password: "XieS0gMFB9NIUKdMiI91"
elasticsearch.ssl.certificate: certs/wildcard_cert.crt
elasticsearch.ssl.key: certs/wildcard_cert.key
elasticsearch.ssl.certificateAuthorities: [ "certs/wildcard_cert.ca.crt" ]
elasticsearch.ssl.verificationMode: certificate

Kibana log:

  log   [09:03:57.842] [info][plugins-system] Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]
  log   [09:03:57.889] [error][data][elasticsearch] Request error, retrying GET https://node-syst.sigmaestimates.com:9200/_xpack => unable to get issuer certificate
  log   [09:03:57.983] [error][admin][elasticsearch] Request error, retrying GET https://node-syst.sigmaestimates.com:9200/.kibana => unable to get issuer certificate
  log   [09:03:57.999] [warning][data][elasticsearch] Unable to revive connection: https://node-syst.sigmaestimates.com:9200/
  log   [09:03:58.014] [warning][data][elasticsearch] No living connections
  log   [09:03:58.014] [warning][licensing][plugins] License information could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections
  log   [09:03:58.014] [error][admin][elasticsearch] Request error, retrying GET https://node-syst.sigmaestimates.com:9200/.kibana_task_manager => unable to get issuer certificate
  log   [09:03:58.062] [warning][admin][elasticsearch] Unable to revive connection: https://node-syst.sigmaestimates.com:9200/
  log   [09:03:58.062] [warning][admin][elasticsearch] No living connections
  log   [09:03:58.062] [warning][migrations] Unable to connect to Elasticsearch. Error: No Living connections
  log   [09:03:58.108] [warning][admin][elasticsearch] Unable to revive connection: https://node-syst.sigmaestimates.com:9200/
  log   [09:03:58.108] [warning][admin][elasticsearch] No living connections
  log   [09:04:00.606] [warning][admin][elasticsearch] Unable to revive connection: https://node-syst.sigmaestimates.com:9200/
  log   [09:04:00.622] [warning][admin][elasticsearch] No living connections
  log   [09:04:00.653] [warning][admin][elasticsearch] Unable to revive connection: https://node-syst.sigmaestimates.com:9200/
  log   [09:04:00.653] [warning][admin][elasticsearch] No living connections

Elasticsearch log:

[2020-01-07T09:02:45,799][INFO ][o.e.d.DiscoveryModule    ] [node-syst] using discovery type [zen] and seed hosts providers [settings]
[2020-01-07T09:02:47,533][INFO ][o.e.n.Node               ] [node-syst] initialized
[2020-01-07T09:02:47,533][INFO ][o.e.n.Node               ] [node-syst] starting ...
[2020-01-07T09:02:47,939][INFO ][o.e.t.TransportService   ] [node-syst] publish_address {10.1.0.4:9300}, bound_addresses {[::]:9300}
[2020-01-07T09:02:48,330][INFO ][o.e.b.BootstrapChecks    ] [node-syst] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-01-07T09:02:48,330][INFO ][o.e.c.c.Coordinator      ] [node-syst] cluster UUID [KRcR7wptQd2qpNIG7XYz3g]
[2020-01-07T09:02:50,642][INFO ][o.e.c.s.MasterService    ] [node-syst] elected-as-master ([1] nodes joined)[{node-syst}{YjVR-WUtSJ2POC-HxSVqfA}{c10k4Sg8TnCLJWlRZrxpVg}{10.1.0.4}{10.1.0.4:9300}{dilm}{ml.machine_memory=8588939264, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 10, version: 73, delta: master node changed {previous [], current [{node-syst}{YjVR-WUtSJ2POC-HxSVqfA}{c10k4Sg8TnCLJWlRZrxpVg}{10.1.0.4}{10.1.0.4:9300}{dilm}{ml.machine_memory=8588939264, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-01-07T09:02:50,971][INFO ][o.e.c.s.ClusterApplierService] [node-syst] master node changed {previous [], current [{node-syst}{YjVR-WUtSJ2POC-HxSVqfA}{c10k4Sg8TnCLJWlRZrxpVg}{10.1.0.4}{10.1.0.4:9300}{dilm}{ml.machine_memory=8588939264, xpack.installed=true, ml.max_open_jobs=20}]}, term: 10, version: 73, reason: Publication{term=10, version=73}
[2020-01-07T09:02:51,424][INFO ][o.e.h.AbstractHttpServerTransport] [node-syst] publish_address {10.1.0.4:9200}, bound_addresses {[::]:9200}
[2020-01-07T09:02:51,424][INFO ][o.e.n.Node               ] [node-syst] started
[2020-01-07T09:02:51,752][INFO ][o.e.l.LicenseService     ] [node-syst] license [488a9dde-99fa-44a2-aabb-337aab7b0cf3] mode [basic] - valid
[2020-01-07T09:02:51,752][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-syst] Active license is now [BASIC]; Security is enabled
[2020-01-07T09:02:51,783][INFO ][o.e.g.GatewayService     ] [node-syst] recovered [4] indices into cluster_state
2

I followed this artical. it took me while but it did worked. it was just test though

https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash

may be you need to put full path for all .crt and .key.
that is the only difference I see on my test setup and your kibana config.

plus I didn't had following two line in my kibana.yml file
elasticsearch.ssl.certificate: certs/wildcard_cert.crt
elasticsearch.ssl.key: certs/wildcard_cert.key

4

It would appear that the certs/wildcard_cert.ca.crt file you are using in Kibana is not the actual issuing certificate chain for the certs/wildcard_cert.crt certificate you have configured in Elasticsearch.

Where did these certificates come from, and what is the connection between them?

6

Okay so you were right Tim :slight_smile:

I tried to reissue my certificates and reinstall them on the server. Now Kibana can connect to ES! Hurray!

The only issue now is that I can not connect to Kibana from outside the server.

EDIT
I solved it by changing the server.name to full DNS name and server.host to 0.0.0.0 in kibana.yml