Kibana SSL/Certificate - Service Auto-Restarts & Fails to Start

Hey all - Unsure where to turn with this one.
Running Elasticsearch 7.16.3 and Kibana 7.16.3 on Ubuntu Server 20.04

Elasticsearch is configured and running successfully as a single-node deployment, currently with...

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

...enabled, intended for testing agents in the future.

Kibana was working without issue, as well, until we attempted to configured client/browser HTTPS by enabling the following lines....

server.ssl.enabled: true
server.ssl.certificate: /etc/ssl/certs/test.pem
server.ssl.key: /etc/ssl/private/test.key

The CSR for this cert was generated, with a new key, on the same server using OpenSSL, then signed on a Windows certificate authority.

Thinking that, maybe, it was a permissions issue, we tried moving the cert to a path within /etc/kibana to allow the kibana user access to it, but it failed in the same manner.

When attempting to start the service, it completes without error initially. After a few seconds, the service goes into an auto-restart/activating status, and repeats this loop a few times till it ultimately fails.

I don't see any events being generated in /var/log/kibana/kibana.log that coincide with these systemctl messages, but I do see HTTP {"type":"response"...} events from accessing the server via a browser (prior to enabling the SSL features in kibana.

If I keep a tail -f going on that log file during the startup attempt for the service, there are no events being written to the log file.

As soon as we comment out the three SSL lines in kibana.yml, it starts up without a problem.
Cert configuration issue? Cert file path issue?

Any insight or tips would be awesome, I'm a bit lost.

Sorry for the late response on this one!

I see we had a similar issue in the past here: systemd restart loop, reported as running with no errors · Issue #10895 · elastic/kibana · GitHub

Kibana won't start if it can't read the certificate for some reason, and systemctl will automatically restart Kibana.

That user suggested:

To see the actual issue, you need to use $ sudo journalctl -f -u kibana.service:
{ Error: ENOENT: no such file or directory, open '/path/to/your/CA.pem' ...

  • Have you tried to use journalctl to view the service logs?
  • Have you double-checked the user/group permissions on the cert/key files?
  • Is your key file encrypted? If so, you'll need to configure the password using server.ssl.keyPassphrase. You can check your key file by using this command:
    openssl rsa -check -noout -in /etc/ssl/private/test.key
    

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.