Elasticsearch ignores the `xpack.security.transport.ssl.certificate_authorities` parameter

Hi, It seems like there is some misleading or a bug with the parameters: key, certificate, certificate_authorities
Elasticsearch ignores the xpack.security.transport.ssl.certificate_authorities parameter
The Elasticsearch version is 7.9.2

The Elasticsearch Cluster log output:

[2020-10-02T16:20:17,967][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elasticsearch-02.example.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.XXX.XXX.42:43602, remoteAddress=elasticsearch-01.example.local/10.XXX.XXX.41:9300}
[2020-10-02T16:20:18,967][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elasticsearch-02.example.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.XXX.XXX.42:43604, remoteAddress=elasticsearch-01.example.local/10.XXX.XXX.41:9300}
[2020-10-02T16:20:19,967][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elasticsearch-02.example.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.XXX.XXX.42:43606, remoteAddress=elasticsearch-01.example.local/10.XXX.XXX.41:9300}
[2020-10-02T16:20:20,968][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elasticsearch-02.example.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.XXX.XXX.42:43608, remoteAddress=elasticsearch-01.example.local/10.XXX.XXX.41:9300}

elasticsearch-02.example.local tries to connect to elasticsearch-01.example.local close the connection becouse the client don't trust this server's certificate.

3x nodes (1 master, 2 data) TLS configuration is:

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: "certificate"
xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/elasticsearch.example.local.key"
xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/elasticsearch.example.local.crt"
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: "/etc/elasticsearch/certs/elasticsearch.example.local.key"
xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/elasticsearch.example.local.crt"
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
  • elasticsearch.example.local.crt - PEM certificate is the same for all 3x servers, has CN = elasticsearch.example.local an SAN (subject alternative name) for all cluster domains: DNS:elasticsearch-01.example.local, DNS:elasticsearch-02.example.local, DNS:elasticsearch-02.example.local
  • elasticsearch.example.local.key - The key for the certificate elasticsearch.example.local.crt
  • ca.crt - The Self Signed CA Certificate used to create the certificate elasticsearch.example.local.crt
  • Conections between servers goes by DNS host names and not by IP (you can see it form the cluster.log output)

OpenSSL tests

#Test command with CA:
openssl s_client -connect elasticsearch-02.example.local:9300 -CAfile /etc/elasticsearch/certs/ca.crt

#Results:
depth=1
verify return:1
depth=0 CN = elasticsearch.example.local
verify return:1
#Test command without CA:
openssl s_client -connect elasticsearch-02.example.local:9300

#Results:
depth=0 CN = elasticsearch.example.local
verify error:num=20:unable to get local issuer certificate
verify return:1

The tests with OpenSSL show that the TLS and the DNS configurations are ok
TLS works with certificates and custom self signed CA with no issues
But Elasticsearch ignores the certificate_authorities parameter and won't verify the certificate using custom CA

CA, Certificate + Key (The files were generated for the post, and not in use of any environemntes)

Example CA:

-----BEGIN CERTIFICATE-----
MIIFUTCCAzmgAwIBAgIUCo8L/qZz3Zb2Gp2anVBnC9vH6+swDQYJKoZIhvcNAQEN
BQAwFTETMBEGA1UEAwwKRXhhbXBsZSBDQTAeFw0xODAxMDEwMDAwMDBaFw0yNzEy
MzAwMDAwMDBaMBUxEzARBgNVBAMMCkV4YW1wbGUgQ0EwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQC99tz7lhsootKFrfaCOUK8jo1nbwqy1dE1lRB/oeLW
6EcNqfc/6YVdxqq76PoAK55SGlna6fZslaVtSabY61uAJPUJq1/YHyC+aA95Koyp
JGRwOnZHZi//5qKKGdYD28aURgMJC8rwei/Msczs1K2akvzX+Ff0RDFRH+I979Nu
gtWhDjwrXNAwWKhrcZY89NJ9l1WCTa0N/Y12/B+k4fTyOLOIBIJzMCW3PrbzwbWM
7DbBit4CncjIFFe55bZgTDKve6J9pce5isQ71rCzVo14WlHLg+krYtLFNOQ4LGEj
L8XqER+DMxYrsqfIEoUCJk1hIAv/PeqEAFNa/j+gvxrz0lYSu3GU2M9JzQbAWxVn
C54kZFqibZAf4K5PUT8facZcDpjN3YSaT8e8UYDo+Lgw0QcPD1tKK2fm6k8LlaFl
foofpbO69tI1GRsBflpJNTmjPO1/EPYl6HeDOcRO/+8K/bfqro1fnJwmzxqnnRp/
DRF05b/ZOBsHW1hUzR6Gnsxlu1KezLzdp3H0PD3CeOyHf4hsjVim1XOwdu54sf44
qJmI7grZaPDf8FKBs19Kz+gRFTuADmw2McbLa/GIxwZhaLaF8PGtnyDirENMPCeU
wZpDxEBATkEBsDCxooGiyPfF9hUejKLPXEWFDZAY5OXDoovueacSjqssrk3H08Oh
mQIDAQABo4GYMIGVMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFCrOh21P
CbKJZ+E2KDTNLwyq9aswMFAGA1UdIwRJMEeAFCrOh21PCbKJZ+E2KDTNLwyq9asw
oRmkFzAVMRMwEQYDVQQDDApFeGFtcGxlIENBghQKjwv+pnPdlvYanZqdUGcL28fr
6zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcNAQENBQADggIBAII6tUqi+Qx/5dG0
FuE/j9mBbfMAoQ6tDhb5QTfmRzRlkac4SLYT5hK/HdJfB/BAf9h4t/TS5LYYe3ez
8Qyr2C7O3xAI34hy8fiZLTBqs+mIzTxa0M7Q0E9mx/gBUa35jlBX71eOW9xpkVYD
/1ic8LJZf3RYofvBly92wNrz5zaT7YDfFLy7S5LdcKJDR5WGIdOQFM81rrTtIqky
g2DbQ9MwshiapSkilfe7X7NzO5dvK9+cOv/fut+KrQbUIKtIxJ882lyLSaSv54g8
eChJq5ZX5UKC8yFKl+daB+l3tn9OMqCl64PWqVKlH6cS09i+OYP+ahWNK4ErO39b
FQUJa8KgznYd3xNhvms9wvjNQkmMRL+s0YsBhE4W92ooFxYg+QeM4lJZ0adRi//9
3WzTbyPI4bdF3bv/oVhB2a3SrWGMRLvKXACR3RwE7bGEJDqTOZja4BEyY+AqavNA
VWNdgI5Y8bZvoVJsZA9fdmVxcn2lEqiGWa+jUg+MXA4+GBRw/p2DxPJpA6qLg2I6
3gBxMQmXlZFyxt+9N0svtzlGhyAU6SaThmft9xpcJbBVRENuDBIlsQgob0/HuWLx
7FNO/hipRrnhQSaChul7t53meYi6JR3RYseMrwj0MQV7SRE2wgSCtRamBt6rZgeR
RN7whg63UQYgMyR6BwPvz9kQXwm+
-----END CERTIFICATE-----

Example Certificate:

-----BEGIN CERTIFICATE-----
MIIG3zCCBMegAwIBAgIEX3dtTDANBgkqhkiG9w0BAQ0FADAVMRMwEQYDVQQDDApF
eGFtcGxlIENBMB4XDTE5MDEwMTAwMDAwMFoXDTI4MTIyOTAwMDAwMFowJjEkMCIG
A1UEAwwbZWxhc3RpY3NlYXJjaC5leGFtcGxlLmxvY2FsMIICIjANBgkqhkiG9w0B
AQEFAAOCAg8AMIICCgKCAgEAtxuS0KkFKrLm3Kvmq5Mb9nH83BCvrQo2iocs4tfS
scJr7PpKRvKuJgT6gWVbjJyiBnFS+4FHocP2iGJauGkQJWPksQ7cxSTFjrjHItif
z5ICowL3uiUony9pDpkn4Jd96coAczsNbuvWBNXnJy+L3QAUcRyk9W27d6X6Kjze
eTn/eTWXf65jdb9b0AGCibgkX7Oe7Tx+mlpY0U2/+yyeMJvfpwIrVGmiBRoJNvwQ
CLM1dQ1atm5ZYaQiH9K8XrjzZnEQqOPHlyNB0IH9dKlWPmcP0BIsZJHnq1h+dnkC
CR5renFOfzuv5udnCP/ZfsH2n2kjPoLDXUqR2FaSe58YKfjTELkfW3v0Bk9+CEJO
r2rqcmER9M1UfaszfNeNbR8v6ytKeik6Gdwa14VJi/aWPhlS9E8IMjXl1iyZ/snJ
ep6ApYcIM3edM9yv49Fo4BAweWvOTX98KglaBssfvcQabunStyYLc4XdMOOutdLn
sIBtn2qVMc8fEZuqlb7l6cOCO8Ui0fadXpqiW1EcD1WRC5RFN5rYFfvBa/qw/UuI
G8q4LrjcCGIWhBi+y/hviCdpZDktrzsJj/RuLuRZ92+OnI8QfAlFj7gNVoOOj95t
BWg870VS555AHtMJOvlLF719KbrIW7AN00GvQQ6TCZ0HkVVNCPyH3Nuc2vLk0MA6
71cCAwEAAaOCAiQwggIgMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFAvlg2y4PVx6
/vZuyOpZy0l+W+laMFAGA1UdIwRJMEeAFCrOh21PCbKJZ+E2KDTNLwyq9aswoRmk
FzAVMRMwEQYDVQQDDApFeGFtcGxlIENBghQKjwv+pnPdlvYanZqdUGcL28fr6zAO
BgNVHQ8BAf8EBAMCBaAwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwEwggF1BgNVHREE
ggFsMIIBaIIbZWxhc3RpY3NlYXJjaC5leGFtcGxlLmxvY2Fsgh5lbGFzdGljc2Vh
cmNoLTAxLmV4YW1wbGUubG9jYWyCHmVsYXN0aWNzZWFyY2gtMDIuZXhhbXBsZS5s
b2NhbIIeZWxhc3RpY3NlYXJjaC0wMy5leGFtcGxlLmxvY2Fsgh5lbGFzdGljc2Vh
cmNoLTA0LmV4YW1wbGUubG9jYWyCHmVsYXN0aWNzZWFyY2gtMDUuZXhhbXBsZS5s
b2NhbIIeZWxhc3RpY3NlYXJjaC0wNi5leGFtcGxlLmxvY2Fsgh5lbGFzdGljc2Vh
cmNoLTA3LmV4YW1wbGUubG9jYWyCHmVsYXN0aWNzZWFyY2gtMDguZXhhbXBsZS5s
b2NhbIIeZWxhc3RpY3NlYXJjaC0wOS5leGFtcGxlLmxvY2Fsgh5lbGFzdGljc2Vh
cmNoLTEwLmV4YW1wbGUubG9jYWyCCWxvY2FsaG9zdDANBgkqhkiG9w0BAQ0FAAOC
AgEAeltTI++EumwL3LImZGyR9diQnmk9rMUpppfTjrs9l++F86L3Qy5CgupUA0lA
k1dgEe6gAzUFa6q/Cc+7OPHcYsiPgOQknjT552tK8qFNn0yziBQSLDNmKN6Tx6nh
2Ru8/7Hvm++7c41m9ZAhQXQVAMkG00bFY7p2LWm5l3XHe5S7wwQq2T4p8GY1E9E+
sxVyHzn3kKf7KnQej0VVysL40LAe5kzMjM5DcXWDmunA9aEtfBJBRw4EpsimMKdi
NKBmqwHAheERGnVA7Ud2qznDaHOOI2yRQjzieQMtj8tcotIySicORqgFMMShkzOG
YweUo+9MxsUgF3JfVbMENQvhOJa1tmsn7EDj//EBhHworJEjlRcaHyLGrWlTlr79
ucwXzQ+bI6y2tjYtwt07kGEDanV6FdnZgLzv4c+aeguTxhoLee+T8+o3iASy/tY6
94QOAAak/s6Z4g0NRCnsx2DMVc7IL4++rXQqlAsboulMzoxcj0ei1t8yywsn92Rk
YFt7yTPd3dMfGzWn4g0ALQzVjBN15rV2pTH4QQwklqZFkMPQL/UBsPcU4zxklT9O
LNex1WcPWW7cIrFMIBJKY3aA1qUfLZpQEbA1DDGvAC2Gu7RZ+cmMN1MR6hq5UpaE
euqKX0VnvjesElPgnsB1UqZxd2tzffHmW6hOrMWT4fM6yrY=
-----END CERTIFICATE-----

Example Certificate Key:

-----BEGIN PRIVATE KEY-----
MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQC3G5LQqQUqsubc
q+arkxv2cfzcEK+tCjaKhyzi19Kxwmvs+kpG8q4mBPqBZVuMnKIGcVL7gUehw/aI
Ylq4aRAlY+SxDtzFJMWOuMci2J/PkgKjAve6JSifL2kOmSfgl33pygBzOw1u69YE
1ecnL4vdABRxHKT1bbt3pfoqPN55Of95NZd/rmN1v1vQAYKJuCRfs57tPH6aWljR
Tb/7LJ4wm9+nAitUaaIFGgk2/BAIszV1DVq2bllhpCIf0rxeuPNmcRCo48eXI0HQ
gf10qVY+Zw/QEixkkeerWH52eQIJHmt6cU5/O6/m52cI/9l+wfafaSM+gsNdSpHY
VpJ7nxgp+NMQuR9be/QGT34IQk6vaupyYRH0zVR9qzN8141tHy/rK0p6KToZ3BrX
hUmL9pY+GVL0TwgyNeXWLJn+ycl6noClhwgzd50z3K/j0WjgEDB5a85Nf3wqCVoG
yx+9xBpu6dK3Jgtzhd0w46610uewgG2fapUxzx8Rm6qVvuXpw4I7xSLR9p1emqJb
URwPVZELlEU3mtgV+8Fr+rD9S4gbyrguuNwIYhaEGL7L+G+IJ2lkOS2vOwmP9G4u
5Fn3b46cjxB8CUWPuA1Wg46P3m0FaDzvRVLnnkAe0wk6+UsXvX0pushbsA3TQa9B
DpMJnQeRVU0I/Ifc25za8uTQwDrvVwIDAQABAoICAAeTxIeN9wDFVnhbRLhk5gtU
vDn/FsX6XjdtUJYyiC2C9iRgVyKIeFxqaFNidl9jO/E4T07JEsXG7jTGnSp0bo6d
i6EoFEq0MtitHNB8VmCokiNcxBOX5g5wT9Ci1dudaUB7c3lRQEUOzVR15ZLhjNAg
aLglj034tIFHBQsqaapqk//YD+Tguozs1kfNSDGaxSaQsErw7exFQNeoeAFU31bz
nFW06UYc+O9tHGSCGcfCYrvtSYuzFU/NEvxuJA1naesbUq0qf9/pPZf7SRnCyCbc
7PKXTe77erLEfrc4WrU15Qy0E6OAPM85VDIzJDTyodoG58Z2cs8xfseNDZLACqfg
DRz74SqBYBOA7KSyJoJRwetsoOJVmeW1WxGcAYsIvOVzXFUlfmPwj8+R/BVDrR/7
gqM8SvtxxLXLSXuweEprFg7iEoTPXskaZtyM3NP1ESIh01OM8HGlC6AmW6aXaEml
XmLeju1tDEtQugW6BLnH42K9Z9RiQE/mW7Yb5xDNtOp4ULYZrVx+v9YJ4a6VoSCp
6i+1rwKncdinuXDEjHd/S9ZpF03N/6Uqz/OHcI44Kniq1HFxtjrYGZY/ActU3Xjk
wHtF0yrkReCtkmIei3HVeqenxCw4raVe8qU6CiCg1FjrP2PkqF2yVk7f0SNdi0cz
KexacUc8RMdCfPEAKLWBAoIBAQDhbZbv73/iqR+lJg/q3yKdPNPGWfLzfqo8bdzc
aq6purW8a1/Mvwm2Z6UIhAiN7SMO6PiFgUMEpk/mcFlcUGQR0DwnAzygkPIdKDk6
5bE4kXfIBLC4x+7ZAG0VGUf3vjfDFTpZcWsF9Wb5vlQsd2ZdTB4IrBw774+20i1W
N12oTGVEweY/T1Hh8/1uXM1OedW33AyoX8etGwoIzVwTbrOe5V7tKrJ+6l8zEyHr
KSalAjazF/ymU0u/EYeHBMFqN6nFUQv6ABIoJhRorxkLFG02DjfyndlfYCHZtTtv
IRmQraBsHwmL6yZFBTnR7kmEdv1kVsZEEXnms1MhknvBtg55AoIBAQDP8LSYRqpF
+yYBLGUy4qr8GR5dJNk3E2/SYUk5hEAhpDMksoAiwShpT4Da0mwFgj86fz8Y/MOz
HxgFSXWrBUaPqi83G7l+Wr+/iIU7w0tcDJSd/HS6Po8/tf63LULp6td3ih0ummmK
ULjTiRH6B7f0lAq/Tty9yVuBlAWE47g12zOl1h7Xczmko1euqtMVd0bxSUwvwkeo
FuRwjwFQhoYFwCYskbkRemHVVDjVwzQhrVRKsx9/mSPk5X6Hnj2y83qB4Kmcok7L
fRFy8BtdA/sI+uFiNE04qEpSb4uOwYdV3/whUo7tXJnaTci8Q8oJlu0yjhgE9Rht
W/ttPl5UcjhPAoIBAENNIpjzmeyjrWzy1iG7CGyFMnyvchc8I7ry2dN8jjTiJ0vf
irRn/JBWlTDbqmok+CiFaV0CYG3M8WHM/QDQh2/CCZNkJf+SBpap4Zwx4x0OYQJ4
VAJ2vlpPBEIeUBwo0CEP/2GguOEIiPva1OqJoygLh5PzPmKufOuKCDsLQIz0ce97
LDCOj9WzsF/4mNrrCOFhcu352nVV80L5fJnYnQu9Uv/flXbKpi1rct7wZDczV/1U
FfvaNm0O4LVwuAXsJGVicv7qcoFGDSXL0OA78vcGe7GVasZMvw5uSxY8fZNyIsjk
40bAp6ZHjyGb0JsZo0EHnx3HUeuMci89unD015ECggEAAWcEMMq5N0OcXAGqpQ82
TAD3XXV0KZTt+iNB1St5K0DqP2Mczh8zYJJysgcdR2Q7fO+Cw25g+4JILTDzpDWi
5jElpF+BfsKp7GbFT0Pf7D0Be5dKSj9FxA48K4wYmW9GCL5buAa+k3o75OFVx7Pp
n7Kn5+ZmQrlq7pLK/rxyb5qM3J93EcjrKJsZakoopkppphLyepf6171fyQqN3bcq
jDthhodtEhsbMc8hmv94CIocuWw/2CJ2RYLuuYdtrxlJSDr1qeLGWUT8qyPHa3ct
aEAY6QqlQq6P4cJy1IGlogSitK7T2/H176z6n5ptcGceLmsa6l2kzogCSBhVlog2
rQKCAQAnewMTKJUenk2RWW29zPQkUsYAo8juhKvRf/GFTedg1wAQfYOMF2KqH05p
U4Ur81kCc097vD+DvMEYB/XHMzlV4OMdVhTsf2tVTGGZZ/KIVrxJcLhSSk3w75W2
8s/WTqAv8f6UjFJ1RLfZ5pZ+8wrMcxU+x6fsdL8o3xXHBN3RouDmqrc0QLzaBMpj
MLNcWF5qtoB2hnHW5HALQhhn7A9iuO4ixLHLNusGfSCeKm2hg5mjomQyj9YOEhUo
c3gEwHK8P45IfuDqHAabUTFX16ky7uuYjZsY3NqJip4zFJOjtqFGp+CNiHSBDcHd
5eWivmjD7rAoJrbA0Rfiy3JsU6YN
-----END PRIVATE KEY-----

Please, show where I have a mistake or confirm if it's a bug in the Elasticserach TLS transport (Elasticsearch ignores the xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ] parameter and won’t verify the certificate using the custom CA)

I have found a solution:
The server certificate (in my case: elasticsearch.example.local.crt) should have extended key usage: serverAuth and clientAuth but in my certificates I have used only serverAuth