I am working on changing over the certificates used for communication between elasticsearch nodes, and for communication between other applications and elasticsearch, from self-signed certificates generated by elasticsearch to certificates issued by our internal corporate CA.
I am having a hell of time, and thought I'd come ask in here in case I am doing something obviously wrong.
Frist, the version of Elasticsearch is 7.16.2
Here is what my original (working) elasticsearch.yml configuration file looks like, using the self-signed certificates:
cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/devnode1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/devnode1.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/ca.crt"]
xpack.security.transport.ssl.key: /etc/elasticsearch/devnode1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/devnode1.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/ca.crt"]
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate
Here is what the non-working elasticsearch.yml looks like:
cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/devnode1.dev.company.local.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/devnode1.dev.company.local.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/companyca.crt"]
xpack.security.transport.ssl.key: /etc/elasticsearch/devnode1.dev.company.local.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/devnode1.dev.company.local.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/companyca.crt"]
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate
I am testing the server health using the following curl call:
curl -X GET "https://devnode1.dev.company.local:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty" --insecure --user elastic:XXXXXXX
Using the former configuration, the health check returns fine, under the latter, it returns something like this:
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?wait_for_status=yellow&timeout=50s&pretty]",
"header" : {
"WWW-Authenticate" : [
"Bearer realm=\"security\"",
"ApiKey",
"Basic realm=\"security\" charset=\"UTF-8\""
]
}
}
],
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?wait_for_status=yellow&timeout=50s&pretty]",
"header" : {
"WWW-Authenticate" : [
"Bearer realm=\"security\"",
"ApiKey",
"Basic realm=\"security\" charset=\"UTF-8\""
]
}
},
"status" : 401
}
In the elasticsearch log, I see something like this (I had to stop kibana/logstash/etc. so I can easily differentiate log entries that are only the result of elasticsearch nodes negotiating with each other):
[2023-01-27T12:49:25,698][WARN ][o.e.c.c.ClusterFormationFailureHelper] [devnode1.dev.company.local] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UUUUUUU, IIIIIII, zzzzzzz], have only discovered non-quorum [{devnode1.dev.company.local}{UUUUUUU}{ppppppp}{10.100.2.8}{10.100.2.8:9300}{dim}]; discovery will continue using [10.100.2.9:9300, 10.100.2.10:9300] from hosts providers and [{devnode1.dev.company.local}{UUUUUUU}{ppppppp}{10.100.2.8}{10.100.2.8:9300}{dim}] from last-known cluster state; node term 268, last-accepted version 123353 in term 268
[2023-01-27T12:49:25,742][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.2.8:45208, remoteAddress=devnode3.dev.company.local/10.100.2.10:9300, profile=default}
[2023-01-27T12:49:25,746][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.2.8:35892, remoteAddress=devnode2.dev.company.local/10.100.2.9:9300, profile=default}
It seems clear to me that there is an issue with some difference between the self-signed certificates and the one issued by our internal corporate CA.
When I compare ca.crt to companyca.crt, there are a few differences, I don't know if they are significant.
From ca.crt:
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Subject: CN=Elastic Certificate Tool Autogenerated CA
X509v3 extensions:
X509v3 Subject Key Identifier:
<bunch of hex>
X509v3 Authority Key Identifier:
keyid:<bunch of hex>
X509v3 Basic Constraints: critical
CA:TRUE
From companyca.crt:
Issuer: DC=local, DC=company, CN=Company Corporate Root CA
Subject: DC=local, DC=company, CN=Company Corporate Root CA
X509v3 extensions:
1.3.6.1.4.1.311.20.2:
...C.A
X509v3 Key Usage:
Digital Signature, Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
<bunch of hex>
1.3.6.1.4.1.311.21.1:
...
That is the root CA certificate, I have also tried using the issuer cert in certificateAuthorities setting (which also doesn't work, and results in the same results). Here are the differences in that certificate:
Issuer: DC=local, DC=company, CN=Company Corporate Root CA
Subject: DC=local, DC=company, CN=Company Corporate Issuing CA
X509v3 extensions:
1.3.6.1.4.1.311.21.1:
.....
1.3.6.1.4.1.311.21.2:
..Z.e..>q.D..I.f....n.
X509v3 Subject Key Identifier:
<bunch of hex>
1.3.6.1.4.1.311.20.2:
.
.S.u.b.C.A
X509v3 Key Usage:
Digital Signature, Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Authority Key Identifier:
keyid:<bunch of hex>
X509v3 CRL Distribution Points:
Full Name:
URI:http://revcheck.company.local/crl/Company%20Corporate%20Root%20CA.crl
Authority Information Access:
CA Issuers - URI:http://revcheck.company.local/crl/companyrootm1.company.local_Company%20Corporate%20Root%20CA.crt
I also have PKCS12 versions of the server certificates issued by the internal corporate CA. I have attempted to use the keystore/truststore configuration using these certficates.
In that case, elasticsearch.yml looks like this:
cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.http.ssl.keystore.password: XXXXXXX
xpack.security.http.ssl.keystore.type: PKCS12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.http.ssl.truststore.password: XXXXXXX
xpack.security.http.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.transport.ssl.keystore.password: XXXXXXX
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.transport.ssl.truststore.password: XXXXXXX
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate
I get the same results with this configuration as I do with the other ones.
I hope this is enough information here to tell what the issue is for someone who understands that stuff a bit better than I do.