No living connections - "Unable to get local issuer certificate"

Elastic Stack Kibana
1

I have a wildcard certificate + key from letsencrypt.

These two files work fine with elasticsearch!

But when using these two files with Kibana, I get the error below and Kibana can't connect to elasticsearch:

Request error, retrying\nGET https://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to get local issuer certificate
Request error, retrying\nHEAD https://elasticsearch:9200/.apm-agent-configuration => unable to get local issuer certificate
Request error, retrying\nGET https://elasticsearch:9200/_xpack => unable to get local issuer certificate
Unable to revive connection: https://elasticsearch:9200/
No living connections

Here's my config right now:

Kibana docker-compose

kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0
    hostname: kibana
    environment:
        - SERVER_NAME=kibana.mydomain.com
        - ELASTICSEARCH_HOSTS=https://elasticsearch:9200
        - SERVER_SSL_ENABLED=true
        - SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/cert.pem
        - SERVER_SSL_KEY=/usr/share/kibana/config/certs/key.pem
        - ELASTICSEARCH_SSL_CERTIFICATE=/usr/share/kibana/config/certs/cert.pem
        - ELASTICSEARCH_SSL_KEY=/usr/share/kibana/config/certs/key.pem
        - ELASTICSEARCH_USERNAME=elastic
        - ELASTICSEARCH_PASSWORD=elastic
        - XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED=true
    ports:
        - 5601:5601
    volumes:
        - "/certs/:/usr/share/kibana/config/certs"
    networks:
        - ${DOCKER_NETWORK_NAME}

VerificationMode=none

I can add SSL_VERIFICATIONMODE=none and kibana works and connects to elasticsearch as expected.

Why can elasticsearch use these certs fine, but kibana is having a problem?

Extra info: Elasticsearch config that works

elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
        hostname: elasticsearch
        environment:
            - cluster.name=docker-cluster
            - discovery.type=single-node
            - network.host=0.0.0.0
            - xpack.security.enabled=true
            - xpack.security.transport.ssl.enabled=true
            - xpack.security.transport.ssl.key=certs/key.pem
            - xpack.security.transport.ssl.certificate=certs/cert.pem
            - xpack.security.http.ssl.enabled=true
            - xpack.security.http.ssl.key=certs/key.pem
            - xpack.security.http.ssl.certificate=certs/cert.pem
            - "ELASTIC_PASSWORD=${ELASTICSEARCH_PASSWORD}"
            - "ES_JAVA_OPTS=${ELASTICSEARCH_JAVA_OPTS}"

Note that the above key.pem and cert.pem are exactly the same I'm trying to use with Kibana.

2

I tried taking both X1 certificate and the X3 certificates from the "Chain of Trust" from the letsencrypt website and placed them for elasticsearch with:

- xpack.security.transport.ssl.certificate_authorities=certs/one.pem,certs/two.pem

and then for kibana:

- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES[0]=/usr/share/kibana/config/certs/one.pem
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES[1]=/usr/share/kibana/config/certs/two.pem

This gives me exactly the same error in the kibana logs as before:

unable to get local issuer certificate

I'm still stuck with this after a day and a half.

I don't want to use a self-signed certificate, this would require I use verificationmode=certificate and I want the full protection of full.

Can anyone please help or offer some advice?

3

Another note: I am using the letsencrypt staging environment and I thought this might be the problem.

As a result I took the staging root and intermediate certs from here: https://letsencrypt.org/docs/staging-environment/ and used these in the certificate authorities sections above, but the problem still remains.

Still the same issue though.

Ok I am one step closer. I used curl -v and also the root certificate and I got a different error - it's complaining because https://elasticsearch isn't owned by me - obviously this is an internal docker one. I've been bitten by this before.

I think that kibana / elasticsearch should be showing me this error as well as it looks like it's just using curl under the hood and not giving proper errors that developers can look into.

So with this in mind, it looks like the only way is either use https://elasticsearch.mydomain.com, which would then require elasticsearch to be public but the certificate resolves fine, or use self-signed certificates along with the self-signed cert CA, solely for the communication between kibana and elasticsearch.

4

The solution is:

  • Elasticsearch must use self-signed certificates (and CA) for both http options (api) and transport options (communication in the cluster). Generate this with the elasticsearch cert utility --dns elasticsearch.
  • Kibana must also use this self-signed certificate to talk to https://elasticsearch for ELASTICSEARCH_SSL vars.
  • Kibana http can use wildcard domain certs for the SERVER_SSL options, so https://kibana.mydomain.com works.

Now kibana can resolve https://elasticsearch because it's using the self-signed cert and can use verification mode full because of the --dns option.

As a result, the elasticsearch API can now only be private because it uses a self-signed cert (no valid https://elasticsearch.mydomain.com).

I'm okay with this, as I don't want elasticsearch publicly accessible on a domain name, but I can imagine it's a downside for others.