Replacing Self-Signed Certificates with Corporate CA Certificates

I am working on changing over the certificates used for communication between elasticsearch nodes, and for communication between other applications and elasticsearch, from self-signed certificates generated by elasticsearch to certificates issued by our internal corporate CA.

I am having a hell of time, and thought I'd come ask in here in case I am doing something obviously wrong.

Frist, the version of Elasticsearch is 7.16.2

Here is what my original (working) elasticsearch.yml configuration file looks like, using the self-signed certificates:

cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.security.http.ssl.key: /etc/elasticsearch/devnode1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/devnode1.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/ca.crt"]
xpack.security.transport.ssl.key: /etc/elasticsearch/devnode1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/devnode1.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/ca.crt"]

xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate

Here is what the non-working elasticsearch.yml looks like:

cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.security.http.ssl.key: /etc/elasticsearch/devnode1.dev.company.local.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/devnode1.dev.company.local.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/companyca.crt"]
xpack.security.transport.ssl.key: /etc/elasticsearch/devnode1.dev.company.local.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/devnode1.dev.company.local.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/companyca.crt"]

xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate

I am testing the server health using the following curl call:

curl -X GET "https://devnode1.dev.company.local:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty" --insecure --user elastic:XXXXXXX

Using the former configuration, the health check returns fine, under the latter, it returns something like this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?wait_for_status=yellow&timeout=50s&pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?wait_for_status=yellow&timeout=50s&pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

In the elasticsearch log, I see something like this (I had to stop kibana/logstash/etc. so I can easily differentiate log entries that are only the result of elasticsearch nodes negotiating with each other):

[2023-01-27T12:49:25,698][WARN ][o.e.c.c.ClusterFormationFailureHelper] [devnode1.dev.company.local] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UUUUUUU, IIIIIII, zzzzzzz], have only discovered non-quorum [{devnode1.dev.company.local}{UUUUUUU}{ppppppp}{10.100.2.8}{10.100.2.8:9300}{dim}]; discovery will continue using [10.100.2.9:9300, 10.100.2.10:9300] from hosts providers and [{devnode1.dev.company.local}{UUUUUUU}{ppppppp}{10.100.2.8}{10.100.2.8:9300}{dim}] from last-known cluster state; node term 268, last-accepted version 123353 in term 268
[2023-01-27T12:49:25,742][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.2.8:45208, remoteAddress=devnode3.dev.company.local/10.100.2.10:9300, profile=default}
[2023-01-27T12:49:25,746][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.2.8:35892, remoteAddress=devnode2.dev.company.local/10.100.2.9:9300, profile=default}

It seems clear to me that there is an issue with some difference between the self-signed certificates and the one issued by our internal corporate CA.

When I compare ca.crt to companyca.crt, there are a few differences, I don't know if they are significant.

From ca.crt:

Issuer: CN=Elastic Certificate Tool Autogenerated CA
Subject: CN=Elastic Certificate Tool Autogenerated CA

	        X509v3 extensions:	
            X509v3 Subject Key Identifier:	
                <bunch of hex>
            X509v3 Authority Key Identifier:	
                keyid:<bunch of hex>	
            X509v3 Basic Constraints: critical	
                CA:TRUE

From companyca.crt:

Issuer: DC=local, DC=company, CN=Company Corporate Root CA
Subject: DC=local, DC=company, CN=Company Corporate Root CA

        X509v3 extensions:
            1.3.6.1.4.1.311.20.2:
                ...C.A
            X509v3 Key Usage:
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                <bunch of hex>	
            1.3.6.1.4.1.311.21.1:
                ...

That is the root CA certificate, I have also tried using the issuer cert in certificateAuthorities setting (which also doesn't work, and results in the same results). Here are the differences in that certificate:

Issuer: DC=local, DC=company, CN=Company Corporate Root CA
Subject: DC=local, DC=company, CN=Company Corporate Issuing CA

	        X509v3 extensions:
            1.3.6.1.4.1.311.21.1:
                .....
            1.3.6.1.4.1.311.21.2:
                ..Z.e..>q.D..I.f....n.
            X509v3 Subject Key Identifier:
                <bunch of hex>
            1.3.6.1.4.1.311.20.2:
                .
.S.u.b.C.A
            X509v3 Key Usage:
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Authority Key Identifier:
                keyid:<bunch of hex>
            X509v3 CRL Distribution Points:
                Full Name:
                  URI:http://revcheck.company.local/crl/Company%20Corporate%20Root%20CA.crl
            Authority Information Access:
                CA Issuers - URI:http://revcheck.company.local/crl/companyrootm1.company.local_Company%20Corporate%20Root%20CA.crt

I also have PKCS12 versions of the server certificates issued by the internal corporate CA. I have attempted to use the keystore/truststore configuration using these certficates.

In that case, elasticsearch.yml looks like this:

cluster.name: devcluster
node.name: devnode1.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.roles: [ master, data, ingest ]
thread_pool.write.queue_size: 3000
discovery.seed_hosts: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
cluster.initial_master_nodes: ["devnode1.dev.company.local", "devnode2.dev.company.local", "devnode3.dev.company.local"]
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.security.http.ssl.keystore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.http.ssl.keystore.password: XXXXXXX
xpack.security.http.ssl.keystore.type: PKCS12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.http.ssl.truststore.password: XXXXXXX
xpack.security.http.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.transport.ssl.keystore.password: XXXXXXX
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/devnode1.dev.company.local.pfx
xpack.security.transport.ssl.truststore.password: XXXXXXX
xpack.security.transport.ssl.truststore.type: PKCS12

xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate

I get the same results with this configuration as I do with the other ones.

I hope this is enough information here to tell what the issue is for someone who understands that stuff a bit better than I do.

You need share more logs from the Elasticsearch nodes. From the logs you shared so far, the cluster is not formed yet.

[2023-01-27T12:49:25,698][WARN ][o.e.c.c.ClusterFormationFailureHelper] [devnode1.dev.company.local] master not discovered or elected yet ...

And the reason is likely because the nodes do not trust each other's certificates. Did you update the YAML file with new key/cert/ca for every ES node? It won't really work if you haven't.

After the below log line

[2023-01-27T12:49:25,746][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.2.8:35892, remoteAddress=devnode2.dev.company.local/10.100.2.9:9300, profile=default}

There should be more diagnostic loggings that give out more details. You need share these logs from all nodes (or at least the two nodes that involved in the connection).

Yes, I understand that the Elasticsearch cluster is not begin formed because the nodes don't trust each other.

Yes, I did update the YAML file with the new key/cert/ca for every ES node.

I included the expanded logs from nodes 1 and 2 below.
I am a little confused by this...it looks like it's trying to use the server certificates as client certificates.

[2023-01-27T12:49:25,742][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.232.8:45208, remoteAddress=devnode3.dev.company.local/10.100.232.10:9300, profile=default}
[2023-01-27T12:49:25,746][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode1.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.232.8:35892, remoteAddress=devnode2.dev.company.local/10.100.232.9:9300, profile=default}
[2023-01-27T12:49:26,024][WARN ][o.e.c.s.DiagnosticTrustManager] [devnode1.dev.company.local] failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [1.2.840.113549.1.9.1=#160b4954406f6174692e6e6574,CN=devnode2.dev.company.local,OU=IT,O=Company,L=City,ST=ST,C=US] and fingerprint [16dc451fef3fdd68a1347bd95b1a82d58e7772be]; the certificate is issued by [CN=Company Corporate Issuing CA,DC=company,DC=local] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [bf8d643b658ecda1dc1f63e7f695dab6d91ffd4f] is trusted in this ssl context ([xpack.security.transport.ssl])
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
        at sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:253) ~[?:?]
        at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:154) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:277) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkClientTrusted(DiagnosticTrustManager.java:93) [elasticsearch-ssl-config-7.16.2.jar:7.16.2]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1279) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) [?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) [?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-01-27T12:49:26,026][WARN ][o.e.t.TcpTransport       ] [devnode1.dev.company.local] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.100.232.8:9300, remoteAddress=/10.100.232.9:57420, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:370) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1301) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more
Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
        at sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:253) ~[?:?]
        at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:154) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:277) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkClientTrusted(DiagnosticTrustManager.java:93) ~[elasticsearch-ssl-config-7.16.2.jar:7.16.2]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1279) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more
[2023-01-27T12:49:25,022][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode2.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.232.9:57416, remoteAddress=devnode1.dev.company.local/10.100.232.8:9300, profile=default}
[2023-01-27T12:49:25,022][WARN ][o.e.d.PeerFinder         ] [devnode2.dev.company.local] address [10.100.232.8:9300], node [null], requesting [false] connection failed: [][10.100.232.8:9300] general node connection failure: handshake failed because connection reset
[2023-01-27T12:49:25,023][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [devnode2.dev.company.local] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.100.232.9:44732, remoteAddress=devnode3.dev.company.local/10.100.232.10:9300, profile=default}
[2023-01-27T12:49:25,023][WARN ][o.e.d.PeerFinder         ] [devnode2.dev.company.local] address [10.100.232.10:9300], node [null], requesting [false] connection failed: [][10.100.232.10:9300] general node connection failure: handshake failed because connection reset
[2023-01-27T12:49:25,064][WARN ][o.e.c.s.DiagnosticTrustManager] [devnode2.dev.company.local] failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [1.2.840.113549.1.9.1=#160b4954406f6174692e6e6574,CN=devnode3.dev.company.local,OU=IT,O=Company,L=City,ST=ST,C=US] and fingerprint [3b8b3a4157fd2fca2a138242530ce806492e4399]; the certificate is issued by [CN=Company Corporate Issuing CA,DC=company,DC=local] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [bf8d643b658ecda1dc1f63e7f695dab6d91ffd4f] is trusted in this ssl context ([xpack.security.transport.ssl])
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
        at sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:253) ~[?:?]
        at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:154) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:277) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkClientTrusted(DiagnosticTrustManager.java:93) [elasticsearch-ssl-config-7.16.2.jar:7.16.2]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1279) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) [?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) [?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]

Nodes within a cluster are talking to each other via the transport interface. They are both client and server at the same time. Therefore, the certificate used for the transport interface must not have constraints such as server or client only. You'll need to re-generate the certs to remove the corresponding Extended Key Usage.

I guess this clears it up...I will need to stick with the self-signed certificates.

Our corporate certificate process allows for server certificates tied to specific servers and client certificates that aren't. We don't have a process that allows for server certificates that can also be used as client certificates.

Is there a particular outcome that you are trying to achieve by replacing the certificate used for transport protocol (communication between nodes)?

We don't recommend using corporate certificates for that purpose. One of the primary reasons for having TLS on the transport port is to restrict which other processes can cannot to the port. That is more effective if you have a dedicated CA for the cluster rather than using a CA that is also used for other purposes.

It's really just process related. Like where the responsibility goes for generating/installing new certificates when they expire, etc.

It's easier for me to automate the corporate certificate stuff because there is already a process in place.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.