Kibana Dev Tools not working after enabling TLS

[ckough (CK)]

Hi There,

I'm running ELK 7.6.2.
My ES nodes are 10.10.10.41-43
My Kibana node is 10.10.10.46
After enabling TLS, Kibana Dev Tools does not work. The error is:

ki01 kibana[1916]: {"type":"error","@timestamp":"2020-04-25T04:02:14Z","tags":[],"pid":1916,"level":"error","error":{"message":"getaddrinfo ENOTFOUND l0.10.10.41 l0.10.10.41:9200","name":"Error","stack":"Error: getaddrinfo ENOTFOUND l0.10.10.41 l0.10.10.41:9200\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)","code":"ENOTFOUND"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?path=%2F_cat%2Findices&method=GET","query":{"path":"/_cat/indices","method":"GET"},"pathname":"/api/console/proxy","path":"/api/console/proxy?path=%2F_cat%2Findices&method=GET","href":"/api/console/proxy?path=%2F_cat%2Findices&method=GET"},"message":"getaddrinfo ENOTFOUND l0.10.10.41 l0.10.10.41:9200"}

The relevant elasticsearch.yml config on all 3 ES nodes is:

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/es1.key
xpack.security.http.ssl.certificate: certs/es1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: certs/es1.key
xpack.security.transport.ssl.certificate: certs/es1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt

The relevant kibana.yml is:

elasticsearch.hosts: ["https://l0.10.10.41:9200", "https://10.10.10.42:9200", "https://10.10.10.43:9200"]
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
elasticsearch.username: "kibana"
elasticsearch.password: "Passw0rd"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]

All other aspects of Kibana are working fine. E.g. I can login with elastic user, see my indices coming in, configure indices, etc.

Thanks in advance,
CK

[flash1293 (Joe Reuter)]

It looks like you have a typo there in your first elasticsearch host (l instead of 1): "https://l0.10.10.41:9200" should probably be "https://10.10.10.41:9200"

[ckough (CK)]

Thanks Joe.

I have corrected the error but now I'm running into

{"statusCode":504,"error":"Gateway Time-out","message":"Client request timeout"}

My test query is just GET /_cluster/health

Any clues?

Thanks,
CK

[flash1293 (Joe Reuter)]

Is there anything else in the logs about Kibana not starting up correctly/not being able to reach one of the hosts?

Also, please try to start Kibana with just elasticsearch.hosts: ["https://10.10.10.42:9200"] and elasticsearch.hosts: ["https://10.10.10.43:9200"] and try again, just to check whether the first host is somehow not reachable?

[ckough (CK)]

Hi Joe,

Resolve this but not sure how. I follow your advice to use 42 and then 43 as my ES host and dev tools works fine. Then I switch to 41 and it also works. I then put in my original settings ES cluster and it still works. what the ...

Thanks,
CK