Hello,
I have a weird problem with trying to establish a SSL communication between kibana and an elasticsearch cluster.
Kibana and ES are both 6.2.1.
Es: p18054.srv.pl.test.net
Kibana: p18053.srv.pl.test.net
Es cluster has 3 nodes, but kibana connects to p18054.
Ssl settings elasticsearch.yml of the node to which Kibana is trying to connect to look like this:
xpack.ssl.key: /etc/elasticsearch/certs/p18054/p18054.key
xpack.ssl.certificate: /etc/elasticsearch/certs/p18054/p18054.crt
xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.enabled: true
SSL settings for kibana.yml are :
elasticsearch.username: "elastic"
elasticsearch.password: "changeme"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/p18053/p18053.crt
server.ssl.key: /etc/kibana/certs/p18053/p18053.key
elasticsearch.ssl.certificate: /etc/kibana/certs/p18053/p18053.crt
elasticsearch.ssl.key: /etc/kibana/certs/p18053/p18053.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]
elasticsearch.ssl.verificationMode: full
server.host: "p18053.srv.pl.test.net"
server.name: "p18053"
elasticsearch.url: "https://p18054.srv.pl.test.net:9200"
Now, the certificates were generated using certutil tool, I did it like this:
certutil ca --ca-dn "CN=Woland,OU=Pieklo,DC=test,DC=com" --pem
certutil cert --ca-cert /usr/share/elasticsearch/bin/x-pack/ca/ca.crt --ca-key /usr/share/elasticsearch/bin/x-pack/ca/ca.key --dns p18054.srv.pl.test.net --name p18054 --pem
certutil cert --ca-cert /usr/share/elasticsearch/bin/x-pack/ca/ca.crt --ca-key /usr/share/elasticsearch/bin/x-pack/ca/ca.key --dns p18053.srv.pl.test.net --name p18053 --pem
and so on for the other nodes.
The error I'm getting from kibana is :
{"type":"log","@timestamp":"2018-02-10T16:20:08Z","tags":["error","elasticsearch","admin"],"pid":27568,"message":"Request error, retrying\nHEAD https://p18054.srv.pl.test.net:9200/ => unable to get issuer certificate"}
{"type":"log","@timestamp":"2018-02-10T16:20:08Z","tags":["error","elasticsearch","data"],"pid":27568,"message":"Request error, retrying\nGET https://p18054.srv.pl.test.net:9200/_xpack => unable to get issuer certificate"}
which I find strange since I can do:
[root@p18053 kibana]# curl https://elastic:changeme@p18054.srv.pl.test.net:9200/_cluster/health?pretty
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
BUT
[root@p18053 kibana]# curl --cacert /etc/kibana/certs/ca.crt https://elastic:changeme@p18054.srv.pl.test.net:9200/_cluster/health?pretty
{
"cluster_name" : "mbcluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 7,
"active_shards" : 15,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Output from openssl:
[root@p18053 kibana]# openssl x509 -in /etc/kibana/certs/ca.crt -noout -issuer -subject
issuer= /DC=com/DC=test/OU=Pieklo/CN=Woland
subject= /DC=com/DC=test/OU=Pieklo/CN=Woland
[root@p18053 kibana]# echo | openssl s_client -connect p18054.srv.pl.test.net:9200 2>/dev/null | openssl x509 -noout -issuer
issuer= /DC=com/DC=test/OU=Pieklo/CN=Woland
When setting elasticsearch.ssl.verificationMode to none in kibana, it is able to connect to the cluster.
The ca.crt files are all the same. Is there any obvious reason why this is not working? I can;t understand why kibana is not able to validate the certificate the cluster is showing.
I've tried to set the elasticsearch certificate as ceritficate authorities in kibana like this:
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" , "/etc/kibna/certs/p18054/p18054.crt" ]
but that didn't solve it. It looks like a problem with CA, but the elasticsearch certificate has been generated using the same ca.crt file kibana has set as elasticsearch.ssl.certificateAuthorities parameter.
I'd really apreciate an idea.
