Kibana failed to load CA: PEM routines:get_name:no start line

[rileghft (楊舜宇)]

I setup an Elasticsearch cluster, and the cluster health is green. Elasticsearch cluster can be reached on kibana server, but kibana failed to start, showing PEM error.

Environment:
OS: Almalinux 8.6
Kibana version: 8.2
Elasticsearch version: 8.2

I manually generate CA and other certificates, following the steps of official guild.
Update security certificates with a different CA | Elasticsearch Guide [8.2] | Elastic
After generating http certificate zip, Elasticsearch-ca.pem under kibana directory in the zip is copied to '/etc/kibana/' and modifying config.

# part of /etc/kibana/kibana.yml
elasticsearch.ssl.certificateAuthorities:
  - /etc/kibana/elasticsearch-ca.pem

file: Elasticsearch-ca.pem

-----BEGIN CERTIFICATE-----
<SECRETS>
-----END CERTIFICATE-----

After kibana starts, it gets error, and kibana shuts down.

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-05-19T15:29:55.543-04:00","message":"error:0909006C:PEM routines:get_name:no start line","error":{"message":"error:0909006C:PEM routines:get_name:no start line","type":"Error","stack_trace":"Error: error:0909006C:PEM routines:get_name:no start line\n    at node:internal/tls/secure-context:65:13\n    at Array.forEach (<anonymous>)\n    at setCerts (node:internal/tls/secure-context:63:3)\n    at configSecureContext (node:internal/tls/secure-context:152:5)\n    at Object.createSecureContext (node:_tls_common:117:3)\n    at Server.setSecureContext (node:_tls_wrap:1344:27)\n    at Server (node:_tls_wrap:1203:8)\n    at new Server (node:https:69:3)\n    at Object.createServer (node:https:105:10)\n    at module.exports.internals.Core._createListener (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:543:79)\n    at new module.exports.internals.Core (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:135:30)\n    at Object.module.exports (/usr/share/kibana/node_modules/@hapi/hapi/lib/server.js:23:18)\n    at new <anonymous> (/usr/share/kibana/node_modules/elastic-apm-node/lib/instrumentation/modules/hapi.js:25:24)\n    at createServer (/usr/share/kibana/node_modules/@kbn/server-http-tools/target_node/create_server.js:18:18)\n    at HttpServer.setup (/usr/share/kibana/src/core/server/http/http_server.js:102:53)\n    at HttpService.preboot (/usr/share/kibana/src/core/server/http/http_service.js:74:51)\n    at Server.preboot (/usr/share/kibana/src/core/server/server.js:175:25)\n    at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)"},"log":{"level":"FATAL","logger":"root"},"process":{"pid":22684},"trace":{"id":"7eb185da53d5ab38acc4a46cc7ebcccb"},"transaction":{"id":"b14d366f11754cec"}}

How to solve the problem?

Besides, Elasticsearch-create-enroldlment-token also failed, and it gets error "ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate".
Moreover, elastic user password was resetted every time when I restart Elasticsearch cluster.

[TimV (Tim Vernum)]

@rileghft

Elasticsearch-create-enroldlment-token also failed, and it gets error "ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate".

How did you configure your node? Did you manually configure security or rely on security auto configuration?

@rileghft

Moreover, elastic user password was resetted every time when I restart Elasticsearch cluster.

It sounds like your cluster has transient storage. Are you running in docker? If so you need to make sure that the config and data directories are mounted from somewhere that will persist between container restarts.

[rileghft (楊舜宇)]

@TimV

docker

Thank you for your reply.

1. I installed elasticsearch from repo, and I missed elastic user password and access tokens when it started. As the result, I manually configured security settings. The following steps are what I did:
   a. Install elasticsearch from elasticsearch 8.2 repo on each machine.

   b. Configure systemd and start all elasticsearch instance. I used pdsh to controll all machines parallelly, and I didn't see elasticsearch password when first running.

   c. Modified elasticsearch.yml. Change values of "node.name", "cluster.name", "network.host", "discovery.seed_hosts", "cluster.initial_master_nodes", "node.roles".

   d. After configing, I restart all elasticsearch machines. I found elasticsearch couldn't connect to each other because of certificate problem.

   e. I followed document to generate CA and signed certificates. "elastic-statck-ca.p12" and certificates for every node signed by ca.crt and ca.key.

   f. Then I created http certificates for every nodes and copied "http.p12" to each node

   g. Modify elasticsearch.yml, changing values of ssl keystore.path and truststore to .p12 and "elastic-stack-ca.p12".

   h. Start elasticsearch cluster.

2. I ran elasticsearch on EXSI VM. There are 3 master nodes and 2 data nodes, and they installed on different machine in network subnet 10.0.1.0/24. Master nodes uses local storage, and data nodes uses external mounted 2TB xfs storage as elasticsearch data storage.