Unable to get issuer certificate


I have a weird problem with trying to establish a SSL communication between kibana and an elasticsearch cluster.
Kibana and ES are both 6.2.1.
Es: p18054.srv.pl.test.net
Kibana: p18053.srv.pl.test.net
Es cluster has 3 nodes, but kibana connects to p18054.

Ssl settings elasticsearch.yml of the node to which Kibana is trying to connect to look like this:

xpack.ssl.key:                     /etc/elasticsearch/certs/p18054/p18054.key
xpack.ssl.certificate:             /etc/elasticsearch/certs/p18054/p18054.crt
xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.enabled: true

SSL settings for kibana.yml are :

elasticsearch.username: "elastic"
elasticsearch.password: "changeme"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/p18053/p18053.crt
server.ssl.key: /etc/kibana/certs/p18053/p18053.key
elasticsearch.ssl.certificate: /etc/kibana/certs/p18053/p18053.crt
elasticsearch.ssl.key: /etc/kibana/certs/p18053/p18053.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]
elasticsearch.ssl.verificationMode: full
server.host: "p18053.srv.pl.test.net"
server.name: "p18053"
elasticsearch.url: "https://p18054.srv.pl.test.net:9200"

Now, the certificates were generated using certutil tool, I did it like this:

certutil ca --ca-dn "CN=Woland,OU=Pieklo,DC=test,DC=com" --pem
certutil cert --ca-cert /usr/share/elasticsearch/bin/x-pack/ca/ca.crt --ca-key /usr/share/elasticsearch/bin/x-pack/ca/ca.key --dns p18054.srv.pl.test.net --name p18054 --pem
certutil cert --ca-cert /usr/share/elasticsearch/bin/x-pack/ca/ca.crt --ca-key /usr/share/elasticsearch/bin/x-pack/ca/ca.key --dns p18053.srv.pl.test.net --name p18053 --pem

and so on for the other nodes.

The error I'm getting from kibana is :

{"type":"log","@timestamp":"2018-02-10T16:20:08Z","tags":["error","elasticsearch","admin"],"pid":27568,"message":"Request error, retrying\nHEAD https://p18054.srv.pl.test.net:9200/ => unable to get issuer certificate"}
{"type":"log","@timestamp":"2018-02-10T16:20:08Z","tags":["error","elasticsearch","data"],"pid":27568,"message":"Request error, retrying\nGET https://p18054.srv.pl.test.net:9200/_xpack => unable to get issuer certificate"}

which I find strange since I can do:

[root@p18053 kibana]# curl https://elastic:changeme@p18054.srv.pl.test.net:9200/_cluster/health?pretty
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.


[root@p18053 kibana]# curl --cacert /etc/kibana/certs/ca.crt https://elastic:changeme@p18054.srv.pl.test.net:9200/_cluster/health?pretty
  "cluster_name" : "mbcluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 7,
  "active_shards" : 15,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

Output from openssl:

[root@p18053 kibana]# openssl x509 -in /etc/kibana/certs/ca.crt -noout -issuer -subject
issuer= /DC=com/DC=test/OU=Pieklo/CN=Woland
subject= /DC=com/DC=test/OU=Pieklo/CN=Woland
[root@p18053 kibana]# echo | openssl s_client -connect p18054.srv.pl.test.net:9200 2>/dev/null | openssl x509 -noout -issuer
issuer= /DC=com/DC=test/OU=Pieklo/CN=Woland

When setting elasticsearch.ssl.verificationMode to none in kibana, it is able to connect to the cluster.
The ca.crt files are all the same. Is there any obvious reason why this is not working? I can;t understand why kibana is not able to validate the certificate the cluster is showing.
I've tried to set the elasticsearch certificate as ceritficate authorities in kibana like this:
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" , "/etc/kibna/certs/p18054/p18054.crt" ]
but that didn't solve it. It looks like a problem with CA, but the elasticsearch certificate has been generated using the same ca.crt file kibana has set as elasticsearch.ssl.certificateAuthorities parameter.
I'd really apreciate an idea.

I'm not quite sure why, yet, but if you specify a distinguished name without the OU and DC, like this: certutil ca --ca-dn "CN=Woland" --pem then everything seems to work properly. As soon as I add in the DC or the OU, it's causing issues, I'm still investigating why.

Thanks Brandon! That gives me something to work with, I thought I had something misconfigured.

Hey @slaypl you identified a bug in our certutil that will be fixed in 6.3, thanks for bringing this to our attention!

High five :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.