SSL Error in Kibana logs

Elastic Stack Kibana
1

Hi guys!
I have successfully configured TLS in ELK using a corporate certificates.
Everything works fine:

  • Kibana and Elasticsearch is available by https
  • Authentication in Kibana is ok
  • Certificate is valid

However, SSL errors appear regularly in the logs (see below)
Please tell me what could be the reason?

elasticsearch conf

node.name: <host_name>
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: <host_ip>
http.port: 9200

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/host.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/host.cer
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/host.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/host.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]
xpack.security.http.ssl.client_authentication: optional

kibana conf

server.port: 5601
server.host: <host_ip>
server.name: "<host_name>"
elasticsearch.hosts: ["https://<host_name>:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: ""

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/host.cer
server.ssl.key: /etc/kibana/certs/host.key

elasticsearch.ssl.certificate: /etc/kibana/certs/host.cer
elasticsearch.ssl.key: /etc/kibana/certs/host.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.cer" ]
elasticsearch.ssl.verificationMode: certificate

kibana logs 
Sep 09 13:37:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:37:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:37:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:37:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:38:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:38:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:38:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:38:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:39:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:39:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:39:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:39:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
...
etc.
2

Hi @artobstrel,

Does your browser trust the corporate certificate?
This looks like the client doesn't trust Kibana's certificate, which causes Kibana to log out error messages. We have an issue open for it on GitHub here: https://github.com/elastic/kibana/issues/35004

If this is the case, it's nothing to worry about, but you should add the CA certificate to your system's trust store so your browser will properly establish trust with Kibana. Then you wouldn't see these errors in the logs anymore.

Best,
-Joe

4

Hi @jportner
On the user side everything looks good.

Chrome:

cert_chrome

IE:

cert_ie
cert_ie615×583 8.71 KB

5

This message appears every second even without visiting the kibana website

6

It might be happening because Kibana is attempting to authenticate with Elasticsearch using its certificate, and Elasticsearch doesn't trust Kibana's certificate.

You don't have a PKI realm enabled anyway, so you can remove these lines from your Elasticsearch config:

xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]
xpack.security.http.ssl.client_authentication: optional

and you can remove these lines from your Kibana config:

elasticsearch.ssl.certificate: /etc/kibana/certs/host.cer
elasticsearch.ssl.key: /etc/kibana/certs/host.key

See if that makes the error messages stop.

7

It didn't help.
I have only one server with ELK for which I issued a PEM-certificate and I want to encrypt traffic between it and other servers and clients.

8

That's good, just be aware you don't need a client certificate to encrypt traffic, you only need a server certificate for that. elasticsearch.ssl.certificate is telling Kibana to use that as a client certificate when connecting to Elasticsearch.

Can you do please do the following:

  • stop Kibana
  • enable debug logging (logging.verbose: true in kibana.yml)
  • clear your log file
  • start Kibana
  • leave it running for 1 minute and stop it again
  • post the whole log file here (not just a few lines)

Best,
-Joe

9

kibana log from start to stop:

10

Thanks for sharing. It is not correlated to the Monitoring plugin, which is what I had originally suspected. Your logs show the error message happening every 15s and 45s, where the Monitoring plugin is making calls to Elasticsearch more often than that.

I was able to reproduce the same exact error message locally (not in the same interval though) by:

  1. Enabling SSL in Kibana
  2. Connecting to Kibana via HTTP

A benign error message is emitted when it expects to see an HTTP header (because the server is initiating an HTTPS redirect), and then the client is redirected and continues as expected and the client receives an empty response as expected*.

The problem lies in how Hapi handles SSL/TLS connections and makes that information available to Kibana. We have some code to catch benign error messages like these and re-assign them to a debug log level. Unfortunately, it looks like that code is broken. I've opened an issue here: https://github.com/elastic/kibana/issues/77391

In the meantime, as to why you're experiencing this without any browsers connecting to Kibana: do you have some other service (such as a load balancer) that sits in front of Kibana and attempts to make health checks? Or do you have Metricbeat running against Kibana? Both of those would periodically make requests to Kibana that may be triggering this behavior. As a workaround to stop these messages until we get the bug fixed, I would suggest checking any of those services and ensuring that they are configured to use HTTPS.

Best,
-Joe

* Update: redirect does not happen / has nothing to do with it, that was just my browser trying to be helpful.

11

Thanks @jportner for your participation in resolving my problem. About balancer. We have a IBM Datapower balancer but it has no effect since elastic and kibana are on the same host. Metricbeat and other similar agents that can access to kibana I have not installed.

The question that haunts me is: who requests kibana via http with this frequency?

I tried to enable audit in Elasticsearch, but unsuccessfully: elasticsearch_audit.json file is empty.

12

An SSL certificate error occurs when a web browser can't verify the SSL certificate installed on a site. Rather than connect you, your browser will display an error message, warning you that the site may be insecure. This message will look different depending on two factors. The first is the browser you're using.

13

certificate is ok

14

I'm not familiar with IBM Datapower, but some load balancers will conduct period health checks even in the absence of inbound traffic. Assuming that you have an HTTP profile set up for Kibana in the load balancer, that could be what's happening.

If all else fails, you could stop Kibana, set up a dummy HTTP server (such as http-server) on the same port on the same host, and log out the requests to find out who/what is making HTTP calls to that host.

Regarding Elasticsearch audit logs: that won't help you figure out what is making calls to Kibana. At any rate, from the logs you provided it looks like you are using a Basic license, and audit logging is a feature for Gold+ subscriptions. Unfortunately, using that configuration value without the proper license will have no effect.

15

One more observation.
I was installing ELK from RPM on 2 servers: test and prod.
On the test server I installed version 7.8 , which I then updated to 7.9.
On the production server, I immediately installed version 7.9.
SSL error occurs on test server.

16

can you share logs ? mcdvoice

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.