I'm trying to use ssl certificates created by Let's Encrypt for elasticsearch and kibana version 8.4. Kibana fails at start up with this error message:
Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate
I think my elasticsearch server is configured correctly, and there's just somethign wrong with kibana configurations.
In terms of what I did, I have the following files on my elastic.example.com server:
ls -alh /etc/elasticserver/certs/elastic.example.com/
-rw-r--r-- 1 elasticsearch elasticsearch 1.9K Oct 6 16:41 cert1.pem
-rw-r--r-- 1 elasticsearch elasticsearch 3.7K Oct 6 16:41 chain1.pem
-rw-r--r-- 1 elasticsearch elasticsearch 5.5K Oct 6 16:41 fullchain1.pem
-rw-r--r-- 1 elasticsearch elasticsearch 1.7K Oct 6 16:41 privkey1.pem
And this is my /etc/elasticsearch/elasticsearch.yml
cluster.name: my-application
network.host: elastic.example.com
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
verification_mode: certificate
key: certs/elastic.example.com/privkey1.pem
certificate: certs/elastic.example.com/cert1.pem
certificate_authorities: certs/elastic.example.com/fullchain1.pem
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
key: certs/elastic.example.com/privkey1.pem
certificate: certs/elastic.example.com/cert1.pem
certificate_authorities: certs/elastic.example.com/fullchain1.pem
cluster.initial_master_nodes: ["e2"]
http.host: 0.0.0.0
Everything seems to work fine for elastic.example.com because I can start up and ping the server with these commands:
systemctl restart elasticsearch.service
curl -X GET -u elastic:demopass "https://elastic.example.com:9200/_cluster/health?pretty=true" --cacert /etc/elasticsearch/certs/elastic.example.com/fullchain1.pem
{
"cluster_name" : "my-application",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 45,
"active_shards" : 45,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 26,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 63.38028169014085
}
Next, I move on to the kibana server. I set up a new server called kibana.example.com and I copied over the ssl certificates so that I have these:
ls -alh /etc/kibana/certs/elastic.example.com/
-rw-r--r-- 1 kibana kibana 1.9K Oct 6 16:52 cert1.pem
-rw-r--r-- 1 kibana kibana 3.7K Oct 6 16:52 chain1.pem
-rw-r--r-- 1 kibana kibana 5.5K Oct 6 16:52 fullchain1.pem
-rw-r--r-- 1 kibana kibana 1.7K Oct 6 16:52 privkey1.pem
I confirmed that the certs for elastic.example.com worked from kibana.example.com with this command:
curl -X GET -u elastic:demopass "https://elastic.example.com:9200/_cluster/health?pretty=true" --cacert /etc/kibana/certs/elastic.example.com/fullchain1.pem
{
"cluster_name" : "my-application",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 45,
"active_shards" : 45,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 26,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 63.38028169014085
}
Next I made this /etc/kibana/kibana.yml:
server.port: 5601
server.host: 0.0.0.0
server.publicBaseUrl: "http://kibana.example.com"
server.ssl.enabled: false
elasticsearch.hosts: ["https://elastic.example.com:9200"]
# I also tried changing `fullchain1.pem` to `chain1.pem` or `cert1.pem`
#elasticsearch.ssl.certificate: /etc/kibana/certs/elastic.example.com/fullchain1.pem
#elasticsearch.ssl.key: /etc/kibana/certs/elastic.example.com/privkey1.pem
#elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elastic.example.com/cert1.pem", "/etc/kibana/certs/elastic.example.com/chain1.pem" ]
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elastic.example.com/fullchain1.pem" ]
# break SSL connection by changing `none` to either `certificate` or `full`
elasticsearch.ssl.verificationMode: none
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
If I do a systemctl start kibana.service everything work fine because there are no systemctl errors and I can see kibana in the web browser.
But the moment I change elasticsearch.ssl.verificationMode from none to certificate or full, I get the error mentioned in my subject title. I've tried various combinations of values for elasticsearch.ssl.* but nothing works unless I keep the verificationMode to none. How do I get SSL to work with verificationMode as either certificate or full when using Let's Encrypt or a commercially signed SSL?
