Kibana SSL

[Flaviu]

I am using Kibana over SSL. After a certificate update (Letsencyrpt renew the certificates every 3 months) I started to get an error

{ 
   "type":"error",
   "@timestamp":"2019-11-26T12:52:22Z",
   "tags":[ 
      "connection",
      "client",
      "error"
   ],
   "pid":14228,
   "level":"error",
   "error":{ 
      "message":"140652453390208:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n",
      "name":"Error",
      "stack":"Error: 140652453390208:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"
   },
   "message":"140652453390208:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"
}

With SSL enabled, If I access Kibana over https in my browser I get the correct certificate, and the certificate is valid, the page is loading correctly without any warning or error message.

If I disable the SSL server.ssl.enabled: false the errors disappear from the logs.

I tried to enable verbose logging, but I could not find any additional information to pinpoint the problem. As this error appears every few seconds, I feel the need to repair it instead of just ignore it.

Can anybody help me figure out what is the problem?

[christophilus (Chris Davies)]

Am I correct in understanding that this Kibana instance was working over SSL previously, and only the recent cert upgrade broke it? Is it behind a proxy or something like that?

[christophilus (Chris Davies)]

Also, someone suggested that this might be a cert issue on your Elasticsearch instance, not on Kibana. Did your ES cert refresh work? You can test by briefly setting Kibana to elasticsearch.ssl.verificationMode: none, but I wouldn't run it in that mode for long, since it is basically turning off SSL verification, eliminating the security benefits of SSL.

[Flaviu]

Christopher, yes, Kibana was working before over SSL without any issue. It is not running behind a proxy.

The ES certificate is fresh too, but I don't think this is the problem. I tried elasticsearch.ssl.verificationMode: none and I get the same error. I don't think it is from elasticsearch SSL because if I configure server.ssl.enabled: false (which is the Kibana SSL), the error disappear.

I am sure the certificate is valid because in the browser I see the certificate start date being a few days ago, it is reported as being valid, and the Kibana server was installed a few months back, so It is clear that something happened when the new certificate was issued and installed by Letsencrypt.

I tried restarting the services (elasticsearch and kibana) also, I tried restarting the server all together.

[Magnus_Kessler]

Please confirm first that with server.ssl.enabled: true, you can access Kibana, and that Kibana connects correctly to Elasticsearch. If this is the case, your certificates are likely deployed correctly.

Could there be another client on your network that connects to Kibana's TCP port? In bigger networks, or on the open internet sometimes other applications try to connect to an SSL enabled port with non-SSL protocols, and this may lead to spurious log entries. These are annoying, but harmless.

[Flaviu]

Yes, I can confirm that with server.ssl.enabled: true everything is working as expected besides the error logs.

Regarding your other question, do you think a filebit or metricbeat could do this? Other than this there is nothing I can think of. What is also strange is that errors started to appear after the certificate was updated.

[Magnus_Kessler]

Identifying which other actor on your network is responsible is difficult without further knowledge. If you suspect that it's another known process, I would check for logs of e.g. your running beats. Maybe they make some assumptions about the certificate_authorities that are no longer true.

You could also use packetbeat to sniff the network traffic hitting your kibana instance and create a list of IP addresses that try to access your server. That should allow you to identify hosts that may be responsible for the traffic. Good luck hunting!