Kibana server not ready/SSL wrong version number

Elastic Stack Kibana
1

Hello,
After an OS patch and reboot of our Elasticsearch server, we are getting the "Kibana server not ready yet" when attempting to log in.

The Kibana log shows:
"message": "Unable to retrieve version information from Elasticsearch nodes. write EPROTO C0570796917F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",

Web browser certificate shows as expiring next year, however, I do show recent expiry when using curl to get certificates from Elasticsearch.

 {
    "path" : "certs/http.p12",
    "format" : "PKCS12",
    "alias" : "http",
    "subject_dn" : "CN=entlogtst01.isllc.local",
    "serial_number" : "aef300a6ace6876505b0dd0c448374ebc006d48a",
    "has_private_key" : true,
    "expiry" : "2024-09-21T22:02:21.000Z",
    "issuer" : "CN=Elasticsearch security auto-configuration HTTP CA"
  },
  {
    "path" : "certs/http.p12",
    "format" : "PKCS12",
    "alias" : "http_ca",
    "subject_dn" : "CN=Elasticsearch security auto-configuration HTTP CA",
    "serial_number" : "5ec2ce1d310d9ab0edb5e0759f3aab8aa78bae13",
    "has_private_key" : true,
    "expiry" : "2025-09-21T22:02:20.000Z",
    "issuer" : "CN=Elasticsearch security auto-configuration HTTP CA"
  },

kibana.yml:

server.host: "10.213.174.66"
server.publicBaseUrl: "http://entlogtst01.isllc.local"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/server.crt
server.ssl.key: /etc/kibana/server.key

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file

pid.file: /run/kibana/kibana.pid

xpack.encryptedSavedObjects.encryptionKey: key
xpack.reporting.encryptionKey: key
xpack.security.encryptionKey: key

elasticsearch.hosts: ['https://10.213.174.66:9200']
elasticsearch.serviceAccountToken: token
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1664837235013.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://10.213.174.66:9200'], ca_trusted_fingerprint: fprint}]

Where to start with getting this resolved?

Thanks,
Jamey

2

So I have rebuilt http.p12 using:
/usr/share/elasticsearch/bin/elasticsearch-certutil http

Certs have been moved to /etc/elasticsearch/certs

Keystore updated with password entered during create using:
/usr/share/elasticsearch/bin/elasticsearch-keystore add "xpack.security.http.ssl.keystore.secure_password"

Updated elasticsearch.yml to point to new certs:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/http.p12
truststore.path: certs/http.p12

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["entlogtst01.isllc.local"]

http.host: 0.0.0.0

Now get error:
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] - this is usually caused by an incorrect password; (no password was provided)

Wouldn't the keystore add have resolved this? Is there some other way to tell it the keystore password?

3

I recreated the http.p12 cert again using:
/usr/share/elasticsearch/bin/elasticsearch-certutil http

And recreated the application certs using the new http.p12:
openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -out server.crt -clcerts -nokeys
openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -out server.key -nocerts -nodes

This resolved the issue. Not sure why the first attempt at recreating the http.p12 failed.

Jamey

1 Like