Hello,
I have a question regarding deploying of a cluster of nodes and stuck with an inter-node communications issue. It seems that my mistake and the solution are pretty obvious, but somewhy I can't grasp what exactly is wrong (honestly speaking, I never deployed Elasticsearch before).
So, I have a K8S cluster (bare metal deployment) and I deploy Elasticsearch with the elasticsearch Helm chart that is available at the https://helm.elastic.co repository. Actually I have 3 deployments, they're called elasticsearch-client, elasticsearch-data and elasticsearch-master, each of these deployments has its own set of Helm-processed values.
elasticsearch-master uses the following values:
---
clusterName: "elasticsearch-logging"
nodeGroup: "master"
roles:
master: "true"
ingest: "false"
data: "false"
ml: "false"
remote_cluster_client: "false"
protocol: https
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.client_authentication: required
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: required
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
elasticsearch-data uses the following values:
---
clusterName: "elasticsearch-logging"
nodeGroup: "data"
roles:
master: "false"
ingest: "true"
data: "true"
ml: "false"
remote_cluster_client: "false"
protocol: https
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.client_authentication: required
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: required
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
elasticsearch-client uses the following values:
---
clusterName: "elasticsearch-logging"
nodeGroup: "client"
roles:
master: "false"
ingest: "false"
data: "false"
ml: "false"
remote_cluster_client: "false"
persistence:
enabled: false
protocol: https
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.client_authentication: required
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: required
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
It's kind of a hybrid of these 2 examples:
- helm-charts/elasticsearch/examples/multi at main · elastic/helm-charts · GitHub,
- helm-charts/elasticsearch/examples/security at main · elastic/helm-charts · GitHub.
The keystore (and. at the same time, the truststore) is pretty primitive. I've made it with Elasticsearch-certutil:
elasticsearch-certutil ca --out ~/elastic-stack-ca.p12 --pass ''-
elasticsearch-certutil cert --name elasticsearch --dns elasticsearch --ca ~/elastic-stack-ca.p12 --pass '' --ca-pass '' --out ~/elastic-certificates.p12.
Actually I'm trying to issue certificates with my own intermediate CA, but I have the same problem with my own certificate, so I decided to issue certificate with Elasticsearch-certutil just to make sure that I really can do it with the recommended tool. And here's the result that I get:
MAC Iteration 10000
MAC verified OK
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Bag Attributes
friendlyName: elasticsearch
localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIMnUYi4qtVnECAggA
- - - - - - - 8< - - - - - - -
MERCELESSLY SKIPPED
- - - - - - - >8 - - - - - - -
q4c=
-----END ENCRYPTED PRIVATE KEY-----
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Certificate bag
Bag Attributes
friendlyName: elasticsearch
localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
subject=/CN=elasticsearch
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIVAP5QDEzgpciNfp9kgbCxZw0mOKRJMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIxMTIxMzE0MzQzOVoXDTI0MTIxMjE0MzQzOVowGDEWMBQG
A1UEAxMNZWxhc3RpY3NlYXJjaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAJOXlUUKEPuZHsHvzhWy4Q5VwhhBmm8z6ZHq/+x0bajRxO6vxGsrg4TXzhyb
GHvu5yh6uLMqObJOZ4Z0g9dypAA3UgYUXg6l/Li2aQQqOGlkF0awYrWWR6Mn0rb7
1ZgTyrI3jkP3OU8Vz1KRsDWXPzjwmuUvQw4AF4a4eyi+UVqkM+DcjgmKvg+uJQY6
/4Ca5VlDSjJwMauDEq8izHCEsqMJgWfHLJcaAgfyIgnd9pRFtNxps9orHDGzC+1H
U6LDxH9h7F08ngG0POrTMg7B4rZ4lnOuarj0QoPKd3HHIPDzFDikr/2n/+eHhsjB
lATSVAcOsgKnJSPDhDpgvsLh5S0CAwEAAaNnMGUwHQYDVR0OBBYEFHY+4OCA54AC
FjRaYO8hEqQnTBGuMB8GA1UdIwQYMBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMBgG
A1UdEQQRMA+CDWVsYXN0aWNzZWFyY2gwCQYDVR0TBAIwADANBgkqhkiG9w0BAQsF
AAOCAQEAkR9V8QosOBaYAqaesOHbX0ghPvSxCuhxrgHqoAzk3mFIxWUQOWl/zHgo
Af2nnJ6odWQ5rTPMIzUB85vSBj2lZb2LIfp6PL2ZiV7PKOqMAo/0sKz1FliXEoOj
AX4P6MDZEoaNeowHIEG0v/tYHMThqAzOXLfiCYPjnanQYTXe9s1UzS3jWEeowLUu
n3e922Mm32qQtaVE4Py/J2w1jV9o2i8Jlx/FfzUaMmfTCNhMdcioxIiWYOwymcUv
zTU3kxFqym8ojNLmkD3nAgNV34xPHNlABQL3y0x1cXcEiHMnzPgD654wX1/ot0Ul
30kd7eVSCX0gEqqL/9gIICxEPtyreA==
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
friendlyName: ca
2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=/CN=Elastic Certificate Tool Autogenerated CA
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUKIlzkBKxxPYjUmdGDB1aefkP5vMwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMjExMjEzMTQzMTMxWhcNMjQxMjEyMTQzMTMxWjA0MTIwMAYD
VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK39C3JleGydZtMh4kjdH+6y
92yss/ogwsZRpDL8yNp+pzrO8PUc3HhKuaxaaYCYFIokhFS1neRoZeTgvcJ1+IOI
BOGYLe1nJ6xUJhbr75JlkjOq7XNbGiArEoqzvv6WzmybVfxrT+pvFoz1aigIZaqw
6EL7+IcAUdcUuptIpYefnnCkeCW2p7zjSCERTaEJVOiD+gTbi1jVujTLzNyLpug4
SuWa0/olkG0lFpoP3jIoyMlwvorJuRANMaWXA4SH+vn/vmvZeIZE5gm7FMnOrn6l
oiVFr0tbZZgVO3fdW1gTAie3iz2i8o+sV5/U87gsjz2yHUVZNVjV6qt4UzElJEkC
AwEAAaNTMFEwHQYDVR0OBBYEFJ6R4wn2GlPuu5x8b1ghaCENQHzkMB8GA1UdIwQY
MBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAFwMYu4QFzRf17TLYCnIFWf18zBgKS7HtIcc+yeORzgIfbmC
rcKhVzxDNwEy8dfl8D3zk0t/IbOoBFWzKWJjHK5brICoZ13CQ4qdhWE2GBcQbwRL
9CDlbEN520XQk5UMOMaLy6AwPJNkzNLu8p6HEYiupmC0t4NZCSlYWHoqwmhgX7gD
PY+OLEHYpmNRLIukRrw7OpKcC2dMbTPqeYEMXrttuICvEpwWxfJ+hU4sh2vT1dSs
OPVsanS+9DtAZRgDHG0UiPYAXmAh8wYT5DY6wm5jGXa7CXLuvZqLN/PAB+wJvDDy
Mr5Qu7Y+zG9EFb05vNiNme2YU/kbpMRLMW8MEGo=
-----END CERTIFICATE-----
And when all the nodes start, they try to communicate to each other, but they can't do that, because they don't seem to show their certificates to each other. 
{"type": "server", "timestamp": "2021-12-13T14:47:38,292Z", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-master-0", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:60682}", "cluster.uuid": "p-UpYVI1QK2dPGN0TMF3Lg", "node.id": "yLOUYcjeS1KIKr6Y6oNhoA" ,
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
...}
Would anyone be so kind as to help me to understand what did I do wrong? Is the certificate inappropriate? Should I have added something to the configuration files?
Lots of thanks in advance.