Kibana throws error while connecting to elastic nodes after enabling SSL/https on elastic nodes

Elastic Stack Kibana
1

Hi Experts,

We have a 3 node ELK cluster and the version details are below:-

elasticsearch-6.1.0
kibana-6.1.0

We have kibana installed on another host.

Recently we integrated X-Pack with ElasticSearch and Kibana.Before the trial license expired,we changed it to basic license and the cluster was working fine and Kibana too.

We implemented SSL/TLS on the Elastic cluster by folliwng the link below.The certificates were self signed.

Master link being followed for security:-

https://www.elastic.co/guide/en/x-pack/6.x/ssl-tls.html

Link for enabling TLS for Elastic search:-

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/configuring-tls.html#node-certificates
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/configuring-tls.html#enable-ssl

The certificates are self signed and without password and the same certificate is being used on all the three nodes as the cert is a generic one and not tied to the hostname or DN name.

After this, we noticed that Kibana could not connect to the elastic nodes and so we enabled "elasticsearch.url" in kibana.yml to use the https protocol(Step -7 in the link below):-

https://www.elastic.co/guide/en/kibana/6.1/using-kibana-with-security.html

NOTE:- we have not followed any of the other steps in the link above.We have nt enabled SSL between the browser to kibana.

Errors seen in Kibana that keep on coming continously:-

{"type":"log","@timestamp":"2018-02-06T15:56:00Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}
{"type":"log","@timestamp":"2018-02-06T15:56:02Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"Unable to revive connection: https://hostname:9200/"}
{"type":"log","@timestamp":"2018-02-06T15:56:02Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}
{"type":"log","@timestamp":"2018-02-06T15:56:05Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"Unable to revive connection: https://hostname:9200/"}
{"type":"log","@timestamp":"2018-02-06T15:56:05Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}
{"type":"log","@timestamp":"2018-02-06T15:56:08Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"Unable to revive connection: https://hostname:9200/"}
{"type":"log","@timestamp":"2018-02-06T15:56:08Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}
{"type":"log","@timestamp":"2018-02-06T15:56:10Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"Unable to revive connection: https://hostname:9200/"}
{"type":"log","@timestamp":"2018-02-06T15:56:10Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}

--

Troubleshooting steps tried:-

1)Tried accessing each of the three hadoop nodes over httos from the browser and it works fine after accepting the cert and it works fine.
2)Tried accessing over http and it results into failure thereby confirming that the elastic nodes are accessible over https only and not through http.
3)Tried accessing the elastic nodes from the kibana host using curl command "curl -k -s https://hoatname:9200" and it returns successful.

So essentially, I see that the issue is only with kibana to connect to elastic nodes.

Tried checking the logs for any other errors and the only other warning I see is :-

{"type":"log","@timestamp":"2018-02-06T15:56:00Z","tags":["license","warning","xpack"],"pid":6512,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}

What we are trying to achieve here is that any client accessing elastic nodes should be over https/TLS.

Clarifications:-

1)DO we need to follow any more steps other than enabling TLS/SSL for elastic search nodes and the change in kibana to get https working?
2)Is it mandatory to follow the section "Configure X-Pack monitoring to use encrypted connections" in the link below to get our objective of enabling https for elastic nodes?

Any help would be highly appreciated!

1 Like
2

Any reply team?Need to get this up and running at the earliest.

Appreciate any help!

3

Please be patient. These forums are operated on a best efforts basis, with no commitment to response times.
It is not reasonable to expect a reply within 3 hours of posting.

If you need support with a formal SLA, we offer various subscriptions for that.

1 Like
4

SSL is part of the X-Pack security feature that requires a paid Gold or Platinum license. It is not supported on a Basic license.

1 Like
5

Thank you for the response.

Sure ..I shall wait for a response and not rush through.

Just to reconfirm...So is it because of having a Basic license that Kibana is not able to connect to the Elastic nodes?With a Gold or Platinum license,would this work?
Do we need to follow any additional steps in addition to what is mentioned in my first post to get this to work?

The messages below started appearing in the logs after enabling https in elasticsearch by enabling "xpack.security.http.ssl.enabled: true" in elasticsearch.yml.

{"type":"log","@timestamp":"2018-02-06T15:56:10Z","tags":["warning","elasticsearch","admin"],"pid":6512,"message":"No living connections"}

6

I suspect your problem is that Kibana hasn't been configured to trust the certificate that ES is providing on the HTTP port.

7

Thanks for the response Tim..Appreciate it!..In that case , should the curl command also fail that I am issuing from the kibana node to connect to ELK node?

The cert was generated using the utility "certutil".No passwords were provided and no hostnames etc..
The generated cacert was used to sign the cert for the node:

bin/x-pack/certutil ca

bin/x-pack/certutil cert --ca elastic-stack-ca.p12

It generated two certs namely elastic-certificates.p12 and elastic-stack-ca.p12.

We are using the same cert "elastic-certificates.p12" for all the elastic nodes.

On the kibana side, the only two settings changes for https are:

1)elasticsearch.ssl.certificateAuthorities - Pointed this to both the "elastic-certificates.p12" and "elastic-stack-ca.p12" certificates but still see the same error.

2)elasticsearch.url: "https://hostname:9200"

We still see the same issue.Also noticed the lines below

{"type":"log","@timestamp":"2018-02-06T18:48:04Z","tags":["error","elasticsearch","data"],"pid":13651,"message":"Request error, retrying\nGET https://hostname:9200/_xpack => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2018-02-06T18:48:04Z","tags":["error","elasticsearch","admin"],"pid":13651,"message":"Request error, retrying\nHEAD https://hostname:9200/ => self signed certificate in certificate chain"}

After this , we see the same warning as mentioned earlier

{"type":"log","@timestamp":"2018-02-07T05:30:38Z","tags":["warning","elasticsearch","admin"],"pid":13651,"message":"Unable to revive connection: https://hostname:9200/"}
{"type":"log","@timestamp":"2018-02-07T05:30:38Z","tags":["warning","elasticsearch","admin"],"pid":13651,"message":"No living connections"}

Is there any other setting to trust the certificate in kibana.
The documentation for kibana only mentions about elasticsearch.url and elasticsearch.ssl.certificateAuthorities.

The only difference I see is that the documentation mentions providing the certificate in pem format for the parameter "elasticsearch.ssl.certificateAuthorities" but the certutil tool creates the cert in P12 format which is what we have provided in our configurations.

8

This is the cause of you problem. That kibana setting requires a PEM formatted certificte. You cannot use a PKCS#12 certificate.

9

Thanks Tim for the help.

I have managed to fix the issue.

First I extracted the Trust cert from "elastic-certificates.p12" in der format and the converted the cert into pem format.

I ran into another issue "Hostname/IP doesn't match certificate's altnames" ,since we have not provided any hostname while generating the certs so that we could use the same cert for all elastic nodes.

To resolve this issue,I disabled hostname validation by providing "xpack.monitoring.elasticsearch.ssl.verificationMode to "certificate" instead of "full" in kibana.xml.

2 Likes
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.