Unable to retrieve version information from Elasticsearch nodes

[Technolust (Joe)]

I'm not able to get kibana to present username password continuously shows:

10.141.0.171:5601
Kibana server is not ready yet.

kibana.log shows:

"ecs":{"version":"8.4.0"},"@timestamp":"2022-09-01T13:03:37.022-04:00","message":"Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 10.141.0.171:53038, Remote: 10.141.0.170:9200","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":900},"trace":{"id":"3ade4c23f557a2ddd0ff197d27775d16"},"transaction":{"id":"dc0af279d635b759"}}

I'm able to curl from the kibana server using the kibana_system user.
I copied the elastic cert (/etc/elasticsearch/certs/http_ca.crt) over to the kibana server (/etc/kibana/certs)

[root@rbdckibana01 kibana]# curl --cacert /etc/kibana/certs/http_ca.crt -u kibana_system https://10.141.0.170:9200
Enter host password for user 'kibana_system':

{
  "name" : "rbdcelastic01",
  "cluster_name" : "rbdc-elk-stack",
  "cluster_uuid" : "gyBRmaH2SLCDle39ZSCdTQ",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Any thoughts on what could be causing and how to fix?

[Technolust (Joe)]

ELASTICSEARCH:

cluster.name: rbdc-elk-stack
node.name: rbdcelastic01
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.141.0.170
http.port: 9200

xpack.security.enabled: true
xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["rbdcelastic01"]

http.host: 0.0.0.0

KIBANA:

server.port: 5601
server.host: "10.141.0.171"
server.publicBaseUrl: "http://10.141.0.171:5601"
server.name: "rbdckibana01"
elasticsearch.hosts: ["http://10.141.0.170:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibbles"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file

pid.file: /run/kibana/kibana.pid

[Technolust (Joe)]

When I change the xpack.security.enabled:true to false I'm able to get in.

[leandrojmp (Leandro Pereira)]

Your elasticsearch.yml has SSL enabled for clients:

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

So, if you enable this you need to use HTTPS, but your kibana.yml is configured to use HTTP.

elasticsearch.hosts: ["http://10.141.0.170:9200"]

Try to change it to https.

[Technolust (Joe)]

You are a godsend! That was it... So simple, I don't know why I didn't catch that ...