Unable to retrieve version information from Elasticsearch node

[slashlinux (Petrisor Ciocoiu)]

After I've enabled the xpack security for integrating the Keycloak SSO on Kibana/Elasticsearch, now my kibana says - Kibana server is not ready yet and the kibana logs showing

{"type":"log","@timestamp":"2021-11-09T02:14:17-05:00","tags":["error","savedobjects-service"],"pid":19586,"message":"Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 127.0.0.1:40818, Remote: 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-11-09T02:14:20-05:00","tags":["error","savedobjects-service"],"pid":19586,"message":"Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 127.0.0.1:40826, Remote: 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-11-09T02:14:22-05:00","tags":["error","savedobjects-service"],"pid":19586,"message":"Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 127.0.0.1:40834, Remote: 127.0.0.1:9200"}

My kibana.yml settings:

xpack.security.enabled: true
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
elasticsearch.ssl.certificateAuthorities: config/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

My Elasticsearch.yml

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: certs/elastic-stack-ca.p12
xpack.security.authc.token.enabled: true

#xpack.security.authc.realms.pki1.type: pki


xpack.security.authc.realms:
  saml.saml1:
    order: 2
    idp.metadata.path: saml-elasticsearch-metadata.xml
    idp.entity_id: "http://192.168.100.201:8080/auth/realms/grafana"
    sp.entity_id:  "http://192.168.100.201:5601/"
    sp.acs: "http://192.168.100.201:5601/api/security/v1/saml"
    sp.logout: "http://192.168.100.201:5601/logout"
    attributes.principal: "nameid:persistent

Certs path location:

[root@localhost kibana]# ls /etc/kibana/config/certs/
client-ca.cer

[root@localhost kibana]# ls /etc/elasticsearch/certs/
elastic-certificates.p12  elastic-stack-ca.p12

[root@localhost elasticsearch]# pwd
/usr/share/elasticsearch
[root@localhost elasticsearch]# ls -ltr
total 668
-rw-r--r--  1 root root   2710 Oct  7 17:53 README.asciidoc
-rw-r--r--  1 root root   3860 Oct  7 17:53 LICENSE.txt
-rw-rw-r--  1 root root 628969 Oct  7 17:58 NOTICE.txt
drwxr-xr-x  2 root root      6 Oct  7 18:08 plugins
drwxr-xr-x  2 root root   4096 Nov  8 06:36 bin
drwxr-xr-x  9 root root    121 Nov  8 06:36 jdk
drwxr-xr-x  3 root root   4096 Nov  8 06:36 lib
drwxr-xr-x 61 root root   4096 Nov  8 06:37 modules
-rw-r--r--  1 root root   1057 Nov  8 09:29 saml-elasticsearch-metadata.xml
-rw-------  1 root root   2672 Nov  8 11:56 elastic-stack-ca.p12
-rw-------  1 root root   3596 Nov  8 11:56 elastic-certificates.p12
**-rw-------  1 root root   3837 Nov  8 13:38 client.p12**
**-rw-r--r--  1 root root   1889 Nov  8 13:39 client.key**
**-rw-r--r--  1 root root   1509 Nov  8 13:39 client.cer**
**-rw-r--r--  1 root root   1397 Nov  8 13:39 client-ca.cer**

If I try to run curl with user kibana_system and client.cer/client.key it returns something:

[root@localhost elasticsearch]# curl -k https://localhost:9200/_xpack/security/_authenticate?pretty --key client.key --cert client.cer --cacert client-ca.cer -u kibana_system
Enter host password for user 'kibana_system':
{
  "username" : "kibana_system",
  "roles" : [
    "kibana_system"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "authentication_type" : "realm"
}

I think the authentication from kibana side to Elasticsearch doesn't work as expected.

Thank you for your help

[slashlinux (Petrisor Ciocoiu)]

I resolved the problem:

If the xpack /ssl is enabled I had to change http to https on Elasticsearch.hosts line :

from:

elasticsearch.hosts: ["http://localhost:9200"]

with :

elasticsearch.hosts: ["https://localhost:9200"]

My error logs:

[2021-11-09T08:25:55,521][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54664}
[2021-11-09T08:25:55,524][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54666}
[2021-11-09T08:25:55,526][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54668}
[2021-11-09T08:25:55,528][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54670}

Problem solved.
Thank you