Kibana SSL "Unable to revive connection: https://localhost:9200/"

[derekmizak (DerekM)]

I have elasticsearch and kibana working on the same node in the dev environment.
I have configured TLS in elasticsearch and I can connect to it:

    root@host1:/home/admin# curl -k -0 https://localhost:9200 -u user:password
{
  "name" : "rbl01",
  "cluster_name" : "cluster1",
  "cluster_uuid" : "c_penenenemb9dA",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "shd5679ghjkkd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

My kibana.yml has the following:

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]
elasticsearch.hosts: ["https://localhost:9200"]

When I start kibana it is unable to connect to Elastic and give me the following error:

May 04 21:57:21 rbl01 kibana[21293]: {"type":"log","@timestamp":"2020-05-04T20:57:21Z","tags":["warning","elasticsearch","data"],"pid":21293,"message":"Unable to revive connection: https://localhost:9200/"}
May 04 21:57:21 rbl01 kibana[21293]: {"type":"log","@timestamp":"2020-05-04T20:57:21Z","tags":["warning","elasticsearch","data"],"pid":21293,"message":"No living connections"}
May 04 21:57:21 rbl01 kibana[21293]: Could not create APM Agent configuration: No Living connections

I have tried to diagnose the situation so I have added to kibana.yml the following

elasticsearch.ssl.verificationMode: none

All works fine with it. However - I will need to have TLS working between Kibana and elasticsearch.

Could you help to diagnose, please?

[Marius_Dragomir]

That sounds like the certificates aren't recognized by the host machine. Can you try and add them to the OS certificate store?

[ikakavas (Ioannis Kakavas)]

Please share your entire configuration , and not just small snippets of it as we need to guess the rest and we usually can't

The most probable scenario here is that the certificate you are using for the http layer of Elasticsearch doesn't have localhost in it's SANs so Kibana can't verify that it actually talks to "localhost".

[derekmizak (DerekM)]

@Marius_Dragomir, @ikakavas You both were close. I have found the issue. When I have generated using:

elasticsearch-certutil http

I have added elasticsearch IP address to it and not domains. I have changed in kibana.yml the parameter:
elasticsearch.ssl.verificationMode: none

to:

elasticsearch.ssl.verificationMode: certificate

and it is working now.
Thank you for looking at it

[ikakavas (Ioannis Kakavas)]

Great, as long as you are aware of what certificate means in this context (no hostname verification) and you're happy with this.

Cheers

[derekmizak (DerekM)]

Yes - its Dev only in PRD I will add host to DNS and do it properly.