Hey,
I'm using Elasticsearch with a Javascript proxy, almost exactly the same as the example provided here: https://github.com/elastic/elasticsearch-js.
We have a few proxies around the world and three servers in our cluster, one master server and two data nodes which I'm self-hosting. We've been testing it for a couple of weeks while we prepare to launch the search on our website... so we know everything is set up properly with the cluster and security certificates. However we randomly get this error from the server which we are having a lot of trouble understanding.
{
"error": "ConnectionError",
"message": "Server certificate CA fingerprint does not match the value configured in caFingerprint",
}
And we are connecting with the Javascript client like this:
const { Client } = require('@elastic/elasticsearch')
const client = new Client({
node: 'https://example.com'
auth: { ... },
// the fingerprint (SHA256) of the CA certificate that is used to sign
// the certificate that the Elasticsearch node presents for TLS.
caFingerprint: '20:0D:CA:FA:76:...',
tls: {
// might be required if it's a self-signed certificate
rejectUnauthorized: false
}
})
However the problem is not specifically how we connect, since it works 95% of the time, and then randomly it throws this error in the logs and stops working, and we are having trouble figuring out why? Is it the Javascript proxy that's going to sleep and losing the CA Fingerprint env variable?
It stopped working on our Azure server-less function so I upgraded to always-on NodeJS servers but we're still getting the error... Is the cluster forgetting to check the CA Fingerprint or calculating it incorrectly sometimes?
We calculate the CA Fingerprint as suggested in the docs:
openssl x509 -fingerprint -sha256 -noout -in /path/to/elastic-stack-ca.crt.pem
However I also found that sometimes on localhost that fingerprint doesn't work and I need to calculate it differently:
# Replace the values of 'localhost' and '9200' to the
# corresponding host and port values for the cluster.
openssl s_client -connect localhost:9200 -servername localhost -showcerts </dev/null 2>/dev/null \
| openssl x509 -fingerprint -sha256 -noout -in /dev/stdin
This command generates a different fingerprint that works instead, so is the server changing the way it calculates the fingerprint sometimes and why are they different fingerprints?
You can see these two methods in the official docs.
These show different fingerprints and it seems like the first method is using our custom elastic-stack-ca.crt.pem while the second one is using the http_ca.crt key... but we don't use the http_ca.crt key anywhere in our elasticsearch.yml file:
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
certificate_authorities: [
certs/elastic-stack-ca.crt.pem
]
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
We are currently using Elasticsearch 8.2.0.
Any ideas what could be going wrong or how we can figure this out?
