Kibana unable to connect elastic search using X-pack

Elastic Stack Kibana
1

Hello i am trying to Configuring SSL, TLS, and HTTPS to secure Elasticsearch , i am using x-pack security feature. Elasticsearch and kibana both are runing on same server, EK version 7.3.1

logs error showing : "Unable to revive connection:"

kibana.yml

server.host: "mydomainname"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/server.crt
server.ssl.key: /etc/kibana/certs/server.key
elasticsearch.hosts: ["https://mydomainname:9200"]
elasticsearch.ssl.certificate: /etc/kibana/certs/server.crt
elasticsearch.ssl.key: /etc/kibana/certs/server.key
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/CA.crt
elasticsearch.ssl.verificationMode: certificate

If i am using elasticsearch.ssl.verificationMode: none I am able to connect elasticsearch

2

Hi @Soniya_Rathore,

In the line below "Unable to revive connection:", what's the exact error message?

3 
{"type":"log","@timestamp":"2019-09-25T13:58:03Z","tags":["warning","elasticsearch","admin"],"pid":19193,"message":"No living connections"}
Sep 25 19:28:05 hydra kibana: {"type":"log","@timestamp":"2019-09-25T13:58:05Z","tags":["warning","elasticsearch","admin"],"pid":19193,"message":"Unable to revive connection: https://mydomianname:9200/"}
4

I was looking for an error along the lines of could not verify root certificate - maybe your CA file doesn't have the right permissions set and can't be used correctly from Kibana?

5

That means that kibana doesn't trust the certificate that Elasticsearch is using for TLS on the http layer. The error that @flash1293 refers to, will be printed out in the elasticsearch logs.

elasticsearch.ssl.certificateAuthorities controls which certificates kibana can trust when connecting to elasticsearch over https and as such you need to set the CA certificate that has signed the certificate that elasticsearch is using. The certificate that elasticsearch is using for TLS on the http layer is what you have configured in

xpack.security.http.ssl.certificate

or

xpack.security.http.ssl.keystore.path

If you are using a keystore and not PEM files.

If the above is not helpful enough, share your elasticsearch.yml file and we'll make concrete suggestions on how to configure kibana too.

6

elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: mydomain
http.host:  mydomain
http.port: 9200
transport.host: mydomainname
discovery.seed_hosts: ["mydomainname"]
cluster.initial_master_nodes: ["mydomainname"]
node.master: true
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/server.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/server.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/CA.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/server.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/server.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/CA.crt
7

Can you share with us how you created /etc/elasticsearch/certs/server.crt and also share the elasticsearch logs ?

Does it also work if you set

elasticsearch.ssl.verificationMode: certificate

in kibana.yml ?

8

This is Enterprise CA generated certificates, i am using same certificates for both kibana and elasticsearch as both are running on same node.

When i am using
elasticsearch.ssl.verificationMode: certificate
it is showing Kibana server is not ready yet, there is no error logs in elasticsearch log

9

Then the most probable issue is that your CA.crt doesn't contain all the intermediary CA certificates that are needed to build the chain of trust between the certificate and the CA certificate. The logs from elasticsearch would help us if you shared them with us. You need to ask the relevant people in your organization to get you the full CA chain as a PEM file and then you can use that to replace CA.crt in your configuration in kibana and elasticsearch

This is almost never a good practice, but this isn't what causes the issue here.

Where is that shown, what are the logs.

we asked you for logs when you get the error we try to troubleshoot. But you made changes first so now we are troubleshooting another issue, and it is hard to get to a resolution if we keep doing that :slight_smile:

10

I didn't make any changes issue is still the same when I put " elasticsearch.ssl.verificationMode: none" in kibana.yml it is working and I can open kibana in the browser, but when I put " elasticsearch.ssl.verificationMode: certificate" it is showing error "Unable to revive connection:" in the log file and in the browser it is showing Kibana server is not ready yet

elasticsearch log

         [2019-09-26T15:53:37,132][INFO ][o.e.c.m.MetaDataIndexTemplateService] [domainname] adding template [.management-beats] for index patterns [.management-beats]

        [2019-09-26T15:53:41,794][INFO ][o.e.c.m.MetaDataIndexTemplateService] [domainname] adding template [.kibana_task_manager] for index patterns [.kibana_task_manager]

        [2019-09-26T15:53:42,886][INFO ][o.e.c.m.MetaDataCreateIndexService] [domainname] [.kibana_task_manager] creating index, cause [auto(bulk api)], templates [.kibana_task_manager], shards [1]/[1], mappings [_doc]

        [2019-09-26T15:53:42,898][INFO ][o.e.c.r.a.AllocationService] [domainname] updating number_of_replicas to [0] for indices [.kibana_task_manager]

        [2019-09-26T15:53:44,775][INFO ][o.e.c.r.a.AllocationService] [domainname] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_task_manager][0]] ...]).

        [2019-09-26T15:53:54,016][INFO ][o.e.c.m.MetaDataCreateIndexService] [domainname] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]

        [2019-09-26T15:53:54,018][INFO ][o.e.c.r.a.AllocationService] [domainname] updating number_of_replicas to [0] for indices [.kibana_1]

        [2019-09-26T15:53:54,186][INFO ][o.e.c.r.a.AllocationService] [domainname] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).

        [2019-09-26T15:53:54,566][INFO ][o.e.c.m.MetaDataMappingService] [domainname] [.kibana_1/9fiqdHt9RGyDIECc_0lICg] update_mapping [_doc]

        [2019-09-26T15:56:58,983][INFO ][o.e.c.m.MetaDataIndexTemplateService] [domainname] adding template [.management-beats] for index patterns [.management-beats]
11

Please take some time to read through the previous message. I shared a lot more than this.

TBC, we need to look at the logs of elasticsearch when elasticsearch.ssl.verificationMode is not set at all ( so the value is the default one ( full ) . This will show what the error is, but the issue is probably what I already shared with you above

12

The CA.crt which i am using is CA chain certificate,also tried with CA chain as a PEM still not working.

After setting elasticsearch.ssl.verificationMode: full following log error

Elasticsearch error

2019-09-26T19:35:52,319][INFO ][o.e.b.BootstrapChecks    ] [mydomain] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-26T19:35:52,344][INFO ][o.e.c.c.Coordinator      ] [mydomain] cluster UUID [rYru6R--SDSdhV5mfvq0QQ]
[2019-09-26T19:35:52,684][INFO ][o.e.c.s.MasterService    ] [mydomain] elected-as-master ([1] nodes joined)[{mydomain}{PyhCFUo5QqG8GNzBVWYLjg}{m2IWSYwAQEWI08pGAlBlmA}{mydomain}{172.17.120.125:9300}{dim}{ml.machine_memory=1820717056, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 32, reason: master node changed {previous [], current [{mydomain}{PyhCFUo5QqG8GNzBVWYLjg}{m2IWSYwAQEWI08pGAlBlmA}{mydomain}{172.17.120.125:9300}{dim}{ml.machine_memory=1820717056, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-09-26T19:35:53,194][INFO ][o.e.c.s.ClusterApplierService] [mydomain] master node changed {previous [], current [{mydomain}{PyhCFUo5QqG8GNzBVWYLjg}{m2IWSYwAQEWI08pGAlBlmA}{mydomain}{172.17.120.125:9300}{dim}{ml.machine_memory=1820717056, xpack.installed=true, ml.max_open_jobs=20}]}, term: 3, version: 32, reason: Publication{term=3, version=32}
[2019-09-26T19:35:53,459][INFO ][o.e.h.AbstractHttpServerTransport] [mydomain] publish_address {172.17.120.125:9200}, bound_addresses {172.17.120.125:9200}
[2019-09-26T19:35:53,461][INFO ][o.e.n.Node               ] [mydomain] started
[2019-09-26T19:35:54,443][INFO ][o.e.l.LicenseService     ] [mydomain] license [b1ed168a-9878-45cf-b261-af76b6bffc58] mode [basic] - valid
[2019-09-26T19:35:54,446][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [mydomain] Active license is now [BASIC]; Security is enabled
[2019-09-26T19:35:54,479][INFO ][o.e.g.GatewayService     ] [mydomain] recovered [3] indices into cluster_state
[2019-09-26T19:35:56,586][INFO ][o.e.c.r.a.AllocationService] [mydomain] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0], [.kibana_1][0], [.kibana_task_manager][0]] ...]).

kibana log error

Sep 26 19:40:01 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:01Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"No living connections"}

Sep 26 19:40:01 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:01Z","tags":["warning","task_manager"],"pid":22412,"message":"PollError No Living connections"}

Sep 26 19:40:04 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:04Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"Unable to revive connection: https://mydomain:9200/"}

Sep 26 19:40:04 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:04Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"No living connections"}

Sep 26 19:40:04 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:04Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"Unable to revive connection: https://mydomain:9200/"}

Sep 26 19:40:04 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:04Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"No living connections"}

Sep 26 19:40:04 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:04Z","tags":["warning","task_manager"],"pid":22412,"message":"PollError No Living connections"}

Sep 26 19:40:06 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:06Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"Unable to revive connection: https://mydomain:9200/"}

Sep 26 19:40:06 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:06Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"No living connections"}

Sep 26 19:40:07 hydra kibana: {"type":"log","@timestamp":"2019-09-26T14:10:07Z","tags":["warning","elasticsearch","admin"],"pid":22412,"message":"Unable to revive connection: https://mydomain:9200/"}
13

Hi,

Apologies, but I don't follow you. If the CA.crt you are using, already contains the whole chain, how did you then also try with "the CA chain as a PEM" ?

Same error ?Different error ? What happens ?

Your ES logs stop at

[2019-09-26T19:35:56,586]

but your kibana errors are from

Sep 26 19:40:07

.
Is there nothing in ES around the same time ?

14

Getting this error

"Request error, retrying\nGET https://domainname:9200/_xpack => connect ECONNREFUSED 172.17.120.125:9200"

15

Please put in some more time into your answers when interacting in these forums. You consistently reply with short answers out of context when asked a few questions in the previous post. This makes it hard for people that try to assist you because they have to write follow up posts and ask the same questions again and then ask for clarifications. This also affects you as it takes more time for you to get a proper answer and help with your problem.

You say you are getting this error. When do you get this error ? Does it answer one of the questions of the previous post or is this a new error ? How about the rest of the questions in the previous post ? Where are the relevant logs ?

1 Like
16

That was certificate issue only. Thank you :slight_smile:

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.