SSL Error in Kibana logs

artobstrel · September 9, 2020, 11:15am

Hi guys!
I have successfully configured TLS in ELK using a corporate certificates.
Everything works fine:

Kibana and Elasticsearch is available by https
Authentication in Kibana is ok
Certificate is valid

However, SSL errors appear regularly in the logs (see below)
Please tell me what could be the reason?

elasticsearch conf

node.name: <host_name>
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: <host_ip>
http.port: 9200

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/host.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/host.cer
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/host.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/host.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]
xpack.security.http.ssl.client_authentication: optional

kibana conf

server.port: 5601
server.host: <host_ip>
server.name: "<host_name>"
elasticsearch.hosts: ["https://<host_name>:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: ""

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/host.cer
server.ssl.key: /etc/kibana/certs/host.key

elasticsearch.ssl.certificate: /etc/kibana/certs/host.cer
elasticsearch.ssl.key: /etc/kibana/certs/host.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.cer" ]
elasticsearch.ssl.verificationMode: certificate

kibana logs

Sep 09 13:37:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:37:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:37:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:37:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:38:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:38:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:38:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:38:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:39:01 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:39:01Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Sep 09 13:39:16 <host_name> kibana[23263]: {"type":"error","@timestamp":"2020-09-09T10:39:16Z","tags":["connection","client","error"],"pid":23263,"level":"error","error":{"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140362884896640:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
...
etc.

jportner · September 9, 2020, 12:45pm

Hi @artobstrel,

Does your browser trust the corporate certificate?
This looks like the client doesn't trust Kibana's certificate, which causes Kibana to log out error messages. We have an issue open for it on GitHub here: https://github.com/elastic/kibana/issues/35004

If this is the case, it's nothing to worry about, but you should add the CA certificate to your system's trust store so your browser will properly establish trust with Kibana. Then you wouldn't see these errors in the logs anymore.

Best,
-Joe

artobstrel · September 9, 2020, 2:12pm

Hi @jportner
On the user side everything looks good.

Chrome:

IE:

cert_ie615×583 8.71 KB

artobstrel · September 9, 2020, 3:32pm

This message appears every second even without visiting the kibana website

jportner · September 9, 2020, 3:36pm

It might be happening because Kibana is attempting to authenticate with Elasticsearch using its certificate, and Elasticsearch doesn't trust Kibana's certificate.

You don't have a PKI realm enabled anyway, so you can remove these lines from your Elasticsearch config:

xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.cer" ]
xpack.security.http.ssl.client_authentication: optional

and you can remove these lines from your Kibana config:

elasticsearch.ssl.certificate: /etc/kibana/certs/host.cer
elasticsearch.ssl.key: /etc/kibana/certs/host.key

See if that makes the error messages stop.

artobstrel · September 10, 2020, 8:03am

It didn't help.
I have only one server with ELK for which I issued a PEM-certificate and I want to encrypt traffic between it and other servers and clients.

jportner · September 10, 2020, 3:10pm

That's good, just be aware you don't need a client certificate to encrypt traffic, you only need a server certificate for that. elasticsearch.ssl.certificate is telling Kibana to use that as a client certificate when connecting to Elasticsearch.

Can you do please do the following:

stop Kibana
enable debug logging (logging.verbose: true in kibana.yml)
clear your log file
start Kibana
leave it running for 1 minute and stop it again
post the whole log file here (not just a few lines)

Best,
-Joe

artobstrel · September 14, 2020, 12:23pm

kibana log from start to stop:

jportner · September 14, 2020, 7:11pm

Thanks for sharing. It is not correlated to the Monitoring plugin, which is what I had originally suspected. Your logs show the error message happening every 15s and 45s, where the Monitoring plugin is making calls to Elasticsearch more often than that.

I was able to reproduce the same exact error message locally (not in the same interval though) by:

Enabling SSL in Kibana
Connecting to Kibana via HTTP

A benign error message is emitted when it expects to see an HTTP header ~~(because the server is initiating an HTTPS redirect), and then the client is redirected and continues as expected~~ and the client receives an empty response as expected*.

The problem lies in how Hapi handles SSL/TLS connections and makes that information available to Kibana. We have some code to catch benign error messages like these and re-assign them to a debug log level. Unfortunately, it looks like that code is broken. I've opened an issue here: https://github.com/elastic/kibana/issues/77391

In the meantime, as to why you're experiencing this without any browsers connecting to Kibana: do you have some other service (such as a load balancer) that sits in front of Kibana and attempts to make health checks? Or do you have Metricbeat running against Kibana? Both of those would periodically make requests to Kibana that may be triggering this behavior. As a workaround to stop these messages until we get the bug fixed, I would suggest checking any of those services and ensuring that they are configured to use HTTPS.

Best,
-Joe

* Update: redirect does not happen / has nothing to do with it, that was just my browser trying to be helpful.

artobstrel · September 15, 2020, 11:37am

Thanks @jportner for your participation in resolving my problem. About balancer. We have a IBM Datapower balancer but it has no effect since elastic and kibana are on the same host. Metricbeat and other similar agents that can access to kibana I have not installed.

The question that haunts me is: who requests kibana via http with this frequency?

I tried to enable audit in Elasticsearch, but unsuccessfully: elasticsearch_audit.json file is empty.

nikkinemo95 · September 15, 2020, 11:52am

An SSL certificate error occurs when a web browser can't verify the SSL certificate installed on a site. Rather than connect you, your browser will display an error message, warning you that the site may be insecure. This message will look different depending on two factors. The first is the browser you're using.

artobstrel · September 15, 2020, 12:03pm

certificate is ok

jportner · September 15, 2020, 7:52pm

I'm not familiar with IBM Datapower, but some load balancers will conduct period health checks even in the absence of inbound traffic. Assuming that you have an HTTP profile set up for Kibana in the load balancer, that could be what's happening.

If all else fails, you could stop Kibana, set up a dummy HTTP server (such as http-server) on the same port on the same host, and log out the requests to find out who/what is making HTTP calls to that host.

Regarding Elasticsearch audit logs: that won't help you figure out what is making calls to Kibana. At any rate, from the logs you provided it looks like you are using a Basic license, and audit logging is a feature for Gold+ subscriptions. Unfortunately, using that configuration value without the proper license will have no effect.

artobstrel · September 16, 2020, 8:21am

One more observation.
I was installing ELK from RPM on 2 servers: test and prod.
On the test server I installed version 7.8 , which I then updated to 7.9.
On the production server, I immediately installed version 7.9.
SSL error occurs on test server.

nikkinemo95 · September 16, 2020, 1:15pm

can you share logs ? mcdvoice

system · October 14, 2020, 1:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.