Lets Encrypt not working from console but does from firefox

Elastic Stack Elasticsearch
1

Good Day All.

I have setup a 2-node cluster ith Elasticsearch and kibana.
I took me quite some time on how to get lets-encrypt real certs to work.
The cert now seems to be ok with kibana and Elasticsearch communication.

However, when I installed Metric beat on a server I got this error in the logs which then required me to do more testing

Error dialing x509: certificate signed by unknown authority","service.name":"metricbeat","network":"tcp","address":"elastic1.atstech.co.za:9200","ecs.version":"1.6.0"}

So I found this odd as when I got to https ://elastic1:9200 from firefox I see a verified cert

elastic
elastic788×533 31 KB

Then I ran this in curl.

curl -v https: /elastic1<canpoint a link here>:9200
*   Trying 102.141.211.65:9200...
* Connected to elastic1<cant point a link here> (102.141.211.65) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: 

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

So this makes no sense so I assume I'm doing something wrong.
Here is my section of config for the certs.

xpack.security.http.ssl:
  enabled: true
  verification_mode: certificate
  key: /etc/elasticsearch/ssl/elastic1.atstech.co.za/privkey1.pem
  certificate: /etc/elasticsearch/ssl/elastic1.atstech.co.za/cert1.pem
  #keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: /etc/elasticsearch/ssl/elastic1.atstech.co.za/privkey1.pem
  certificate: /etc/elasticsearch/ssl/elastic1.atstech.co.za/cert1.pem
  certificate_authorities: /etc/elasticsearch/ssl/elastic1.atstech.co.za/fullchain1.pem
  #keystore.path: certs/transport.p12
  #truststore.path: certs/transport.p12

And here are the steps I filled to generate the certs

snap install certbot --classic
certbot certonly -d elastic1.atstech.co.za

===
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/elastic1.atstech.co.za/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/elastic1.atstech.co.za/privkey.pem
===
#note the archive in belwo command

mkdir /etc/elasticsearch/ssl
rsync -avz /etc/letsencrypt/archive/elastic1.atstech.co.za /etc/elasticsearch/ssl/
chmod 750 /etc/elasticsearch/ssl/elastic1.atstech.co.za
chmod 640 /etc/elasticsearch/ssl/elastic1.atstech.co.za/*
chown -R root:elasticsearch /etc/elasticsearch/ssl/elastic1.atstech.co.za

mkdir /etc/kibana/ssl
rsync -avz /etc/letsencrypt/archive/elastic1.atstech.co.za /etc/kibana/ssl/
chmod 750 /etc/kibana/ssl/elastic1.atstech.co.za
chmod 640 /etc/kibana/ssl/elastic1.atstech.co.za/*
chown -R root:kibana /etc/kibana/ssl/elastic1.atstech.co.za

Any help would be greatly appreciated.
Regards

2

Please note that elastic1 all points correctly, however the create topic link doesn't allow me to type it here

3

I am not sure if you edited the curl....
But you should see the Subject Names and SANs do you?

# curl -v -u elastic https://elasticsearch.mydomain.net:9200
Enter host password for user 'elastic':
*   Trying 35.235.78.175:9200...
* TCP_NODELAY set
* Connected to elasticsearch.mydomain.net (35.235.78.175) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=kibana.mydomain.net
*  start date: Dec 11 05:31:33 2023 GMT
*  expire date: Mar 10 05:31:32 2024 GMT
*  subjectAltName: host "elasticsearch.mydomain.net" matched cert's "elasticsearch.mydomain.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: elasticsearch.mydomain.net:9200
> Authorization: Basic asfdgasdfsdfasdfWdYY0ZUenZVbQ==
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 542
< 
{
  "name" : "stephenb-es-8-test",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "BCwYFLXqQLu1THBiiV9oTw",
  "version" : {
    "number" : "8.11.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "6f9ff581fbcde658e6f69d6ce03050f060d1fd0c",
    "build_date" : "2023-11-11T10:05:59.421038163Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host elasticsearch.mydomain.net left intact

I also see this in yours

* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate

That needs to be the fullchain.pem I think you do not have the fullchain in there..

Also

* TLSv1.2 (OUT), TLS header, Unknown (21):

That is old(er) TLS...

certificate: /etc/elasticsearch/ssl/elastic1.atstech.co.za/cert1.pem

I suspect you do not have the fullchain in that cert.

I use Let's Encrypt

I use the TXT challenge method

sudo certbot certonly --agree-tos --manual --preferred-challenges dns --server https://acme-v02.api.letsencrypt.org/directory -d  "elasticsearch.mydomain.net

which produces

fullchain.pem
privkey.pem

then my settings are

xpack.security.http.ssl:
  enabled: true
  certificate: certs/fullchain.pem 
  key: certs/privkey.pem
4

Thank you so much.
It did create the file but I was not pointing it correctly change to use fullcahin1.pem seems to have worked great thank you very much fr your help