Kibana cannot connect to Elasticsearch after enabling SSL/TLS

I am running a two node ES cluster and a single Kibana instance using Docker Swarm,
everything worked fine before enabling SSL/TLS on the ES nodes.

I am using same certificate without hostname verification for all the ES nodes and everything works perfect for the ES nodes. The only problem is that Kibana is not able to connect to ES cluster.
I am using the same certificate for all the ES instances and for Kibana too.

Following is my docker-compose.yml-

version: "3.2"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    restart: always
    environment:
      - node.name={{.Node.Hostname}}
      - discovery.seed_hosts=elasticsearch
      - cluster.name=docker-cluster
      - cluster.initial_master_nodes=node1,node2
      - network.host=0.0.0.0
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms3g -Xmx3g"
      - "ELASTIC_PASSWORD=myespassword"
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/elasticsearch/elasticsearch.key
    secrets:
      - source: es_ssl_key
        target: $CERTS_DIR/elasticsearch/elasticsearch.key
      - source: es_ssl_ca
        target: $CERTS_DIR/ca/ca.crt
      - source: es_ssl_crt
        target: $CERTS_DIR/elasticsearch/elasticsearch.crt
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      mode: replicated
      replicas: 1
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - es_data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:7.2.0
    restart: always
    environment:
      - "SERVER_NAME=kibana"
      - "ELASTICSEARCH_HOSTS=https://elasticsearch:9200"
      - "XPACK_SECURITY_ENABLED=true"
      - "ELASTICSEARCH_USERNAME=kibana"
      - "ELASTICSEARCH_PASSWORD=mykbpassword"
      - elasticsearch.ssl.certificateAuthorities=$CERTS_DIR/ca/ca.crt
      - elasticsearch.ssl.verificationMode=certificate
    secrets:
      - source: es_ssl_ca
        target: $CERTS_DIR/ca/ca.crt
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

secrets:
  es_ssl_key:
    file: ./es_certs/elasticsearch/elasticsearch.key
  es_ssl_ca:
    file: ./es_certs/ca/ca.crt
  es_ssl_crt:
    file: ./es_certs/elasticsearch/elasticsearch.crt

volumes:
  es_data:

I am using same certificate for all the ES nodes and Kibana generated using the following
create-certs.yml-

version: "2.2"
services:
  create_certs:
    container_name: create_certs
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    command: >
      bash -c '
        yum install -y -q -e 0 unzip;
        if [[ ! -f /certs/bundle.zip ]]; then
          bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
          unzip /certs/bundle.zip -d /certs;
        fi;
        chown -R 1000:0 /certs  '
    user: "0"
    working_dir: /usr/share/elasticsearch
    volumes:
      - ./es_certs:/certs
      - .:/usr/share/elasticsearch/config/certificates

instances.yml

instances:
  - name: elasticsearch
    dns:
      - elasticsearch
      - localhost
    ip:
      - 127.0.0.1

I am able to successfully perform a curl call from inside of the Kibana container to the https Elasticsearch endpoint using the following command-

curl --cacert ca.crt -u kibana:mykbpassword https://elasticsearch:9200

However normally Kibana cannot connect to the ES nodes and I receive the following logs-

es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 10.0.3.2:9200"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:xpack_main@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:graph@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:elasticsearch@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_xpack => connect ECONNREFUSED 10.0.3.2:9200"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.bl9aozgf1b0z@node1    | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"}

What happens if you remove the ES user and ES password from Kibana config file?

Receiving the same errors-

es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["status","plugin:xpack_main@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["status","plugin:graph@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_xpack => unable to verify the first certificate"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:17Z","tags":["reporting","browser-driver","warning"],"pid":1,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:17Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:17Z","tags":["status","plugin:reporting@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - No Living connections","prevState":"uninitialized","prevMsg":"uninitialized"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["status","plugin:security@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from green to red - No Living connections","prevState":"green","prevMsg":"Ready"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["warning","telemetry"],"pid":1,"message":"Error scheduling task, received NotInitialized: Tasks cannot be scheduled until after task manager is initialized!"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}

The following lines from the above logs catch my eye-

es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}

es_stack_kibana.1.trlyzkb3sa01@node1    | {"type":"log","@timestamp":"2019-08-07T12:40:16Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_xpack => unable to verify the first certificate"}

Can somebody please help me out, I'm stuck for days.:cry:

So they can reach each other, if you have tried the connection with cURL inside the container?
If the error is "unable to verify the first certificate", then the issue is with those.
I would proceed by starting up the containers and opening a shell to both of them and checking the configs inside containers and double checking certs (both client and CA).

I am using the same certificate for both the ES clusters and Kibana.
ES clusters are able to discover and communicate each other over TLS successfully.

Performing a cURL call from inside the Kibana container to the ES cluster is successful which implies the certificate used for performing cURL is perfect. But Kibana doesn't connect to ES clusters automatically as it should using the same exact certificate.

And the cert is valid for both the dns name and IP address ?

I'm using the following instances.yml to generate the certificates-
Reference- Encrypting communications in an Elasticsearch Docker Container

instances:
  - name: elasticsearch
    dns:
      - elasticsearch
      - localhost
    ip:
      - 127.0.0.1

I am not using a Domain name for my node's IP address.

I always use mutual authentication with certificates, so I am not sure if this setting exists:

xpack.security.http.ssl.verification_mode

If it does, I think you should set it to "certificate" too, because I think it defaults to "full" which doesn't work in your case if you use one cert for both kibana and elasticsearch, and you have only written it for "elasticsearch" and not "kibana".
You can also add "kibana" in the dns list in your instances.yml if you want to see if that is the issue.

I've used the following
elasticsearch.ssl.verificationMode=certificate
as you can see in the above posted docker-compose.yml file.

I found this setting in this Configuring Kibana
and I don't find this xpack.security.http.ssl.verification_mode setting in it.

However I'll try and let you know.

I've used the following
elasticsearch.ssl.verificationMode=certificate
as you can see in the above posted docker-compose.yml file.

Yes, but that is for Kibana. Not Elasticsearch.

I added kibana in the DNS field of instances.yml and this setting xpack.security.http.ssl.verification_mode too.

Still same error-
=> unable to verify the first certificate

If somebody can share their configuration wherein they have successfully connected ES and Kibana over TLS it would be so nice.

I don't understand why the procedures from the documentation doesn't work out of the box.

This error "unable to verify the first certificate" is a waste.
At least there should be some descriptive message to point developers in the correct direction.

I've been reading the docs repeatedly for the past 1 week, going through logs, scraping forums looking for a solution.

Also check Elasticsearch logs if there is an error.
Other than that, I would start deconstructing the config one by one until it works, and then retry.

I have seen the same error before and in that case it was more of a symptom but not the root cause.

I believe your problem here is that you are trying to pass environment variables using a settings syntax.

From: https://www.elastic.co/guide/en/kibana/7.2/docker.html

For compatibility with container orchestration systems, these environment variables are written in all capitals, with underscores as word separators. The helper translates these names to valid Kibana setting names.

Your first 5 variables are in the correct format, but the last 2 have not been converted.

1 Like

Thanks @TimV
The solution was exactly that you suggested.
All my configuration files were perfect, the only wrong thing that I was doing was not passing Kibana environment variables the way it was mentioned in the docs.

Thank you again, saved my day.:heart_eyes: