Hello,
I'd like to sign the nodes' certificates by openssl (instead of Elasticsearch-certutil). Is that ever possible?
When I have a PKCS12 keystore made by Elasticsearch-certutil, everything works like a charm. Here's an example of such keystore:
MAC Iteration 10000
MAC verified OK
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Bag Attributes
friendlyName: elasticsearch
localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
Key Attributes: <No Attributes>
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQI8fg0MhFIdBQCAggA
- - - 8< - - - SKIPPED - - - >8 - - -
BcY=
-----END ENCRYPTED PRIVATE KEY-----
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Certificate bag
Bag Attributes
friendlyName: elasticsearch
localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
subject=/CN=elasticsearch
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIVAP5QDEzgpciNfp9kgbCxZw0mOKRJMA0GCSqGSIb3DQEB
- - - 8< - - - SKIPPED - - - >8 - - -
30kd7eVSCX0gEqqL/9gIICxEPtyreA==
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
friendlyName: ca
2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=/CN=Elastic Certificate Tool Autogenerated CA
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUKIlzkBKxxPYjUmdGDB1aefkP5vMwDQYJKoZIhvcNAQEL
- - - 8< - - - SKIPPED - - - >8 - - -
Mr5Qu7Y+zG9EFb05vNiNme2YU/kbpMRLMW8MEGo=
-----END CERTIFICATE-----
When I provides the nodes with this keystore, they talk to each other and feel pretty happy.
Though I'd like to stick to openssl. Here's the command I run: openssl pkcs12 -export -chain -CAfile ***/ca.cert.pem -in ***/elasticsearch-node.cert.pem -inkey ***/elasticsearch-node.key.pem -out ***/elasticsearch-node.p12, the PKCS12 looks like as follows:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
localKeyID: 02 24 72 54 CA BD 50 16 57 22 D9 A9 5B 87 B9 5B E2 23 03 48
friendlyName: elasticsearch-node
subject=/C=UA/ST=Kyiv City/O=***/OU=***/CN=elasticsearc-node/emailAddress=***@***.***
issuer=/C=UA/ST=Kyiv City/L=Kyiv/O=***/OU=***/CN=*** Root CA/emailAddress=***@***.***
-----BEGIN CERTIFICATE-----
MIIDkjCCAnoCAhABMA0GCSqGSIb3DQEBCwUAMIGXMQswCQYDVQQGEwJVQTESMBAG
- - - 8< - - - SKIPPED - - - >8 - - -
lRQXiTnp
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
friendlyName: ca
subject=/C=UA/ST=Kyiv City/L=Kyiv/O=***/OU=***/CN=*** Root CA/emailAddress=***@***.***
issuer=/C=UA/ST=Kyiv City/L=Kyiv/O=***/OU=***/CN=*** Root CA/emailAddress=***@***.***
-----BEGIN CERTIFICATE-----
MIIEFjCCAv6gAwIBAgIJAOlhXD+eh8KGMA0GCSqGSIb3DQEBCwUAMIGXMQswCQYD
- - - 8< - - - SKIPPED - - - >8 - - -
SI2pwl26FV3o0DAkkJOGt3PkGl9kAYjLJB8neNf3Kp/sB1hVDcY8AA2P
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
localKeyID: 02 24 72 54 CA BD 50 16 57 22 D9 A9 5B 87 B9 5B E2 23 03 48
friendlyName: elasticsearch-node
Key Attributes: <No Attributes>
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIXswZQPqlsrACAggA
- - - 8< - - - SKIPPED - - - >8 - - -
neI=
-----END ENCRYPTED PRIVATE KEY-----
This keystore seems to be unusable, as my nodes can't communicate to each other. 
Here's what the client node say when it tries to establish a connection to one of the master nodes:
{"type": "server", "timestamp": "2021-12-13T22:30:12,870Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-client-0", "message": "master not discovered yet: have discovered [{elasticsearch-logging-client-0}{naaf0ikaTBueM3H4rB2QKw}{unYMPpBkTUSLshihc3LarA}{10.233.110.128}{10.233.110.128:9300}]; discovery will continue using [10.233.103.7:9300, 10.233.116.221:9300, 10.233.108.230:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2021-12-13T22:30:18,018Z", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-client-0", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.110.128:41332, remoteAddress=elasticsearch-logging-master-headless/10.233.116.221:9300, profile=default}], closing connection",
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate",
Here's what the master node say at this very moment:
{"type": "server", "timestamp": "2021-12-13T22:30:18,016Z", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-master-0", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.116.221:9300, remoteAddress=/10.233.110.128:41332, profile=default}], closing connection",
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
So, the client node doesn't use its own certificate when it tries to establish a TLS-protected connection. Something is wrong, but what exactly?
I've got a bad feeling that the CA certificate must have the following attribute: 2.16.840.1.113894.746875.1.1: <Unsupported tag 6>, so it will be considered as trusted by the some TLS library that Elasticsearch uses. But. as far as I understand, openssl won't let me add this attribute (am I right?).
Of course, I considered using PEM certificates instead of a PKCS12 keystore. It might be a good solution too. Though I'd like to figure it out is that possible to use a PKCS12 keystore made by openssl.
Thanks in advance for any hints.