Can't get nodes communicate to each other via TLS-secured transport

v_melnyk · December 13, 2021, 2:53pm

Hello,

I have a question regarding deploying of a cluster of nodes and stuck with an inter-node communications issue. It seems that my mistake and the solution are pretty obvious, but somewhy I can't grasp what exactly is wrong (honestly speaking, I never deployed Elasticsearch before).

So, I have a K8S cluster (bare metal deployment) and I deploy Elasticsearch with the elasticsearch Helm chart that is available at the https://helm.elastic.co repository. Actually I have 3 deployments, they're called elasticsearch-client, elasticsearch-data and elasticsearch-master, each of these deployments has its own set of Helm-processed values.

elasticsearch-master uses the following values:

---

clusterName: "elasticsearch-logging"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "false"
  data: "false"
  ml: "false"
  remote_cluster_client: "false"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

elasticsearch-data uses the following values:

---

clusterName: "elasticsearch-logging"
nodeGroup: "data"

roles:
  master: "false"
  ingest: "true"
  data: "true"
  ml: "false"
  remote_cluster_client: "false"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

elasticsearch-client uses the following values:

---

clusterName: "elasticsearch-logging"
nodeGroup: "client"

roles:
  master: "false"
  ingest: "false"
  data: "false"
  ml: "false"
  remote_cluster_client: "false"

persistence:
  enabled: false

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

It's kind of a hybrid of these 2 examples:

The keystore (and. at the same time, the truststore) is pretty primitive. I've made it with Elasticsearch-certutil:

elasticsearch-certutil ca --out ~/elastic-stack-ca.p12 --pass ''
elasticsearch-certutil cert --name elasticsearch --dns elasticsearch --ca ~/elastic-stack-ca.p12 --pass '' --ca-pass '' --out ~/elastic-certificates.p12.

Actually I'm trying to issue certificates with my own intermediate CA, but I have the same problem with my own certificate, so I decided to issue certificate with Elasticsearch-certutil just to make sure that I really can do it with the recommended tool. And here's the result that I get:

MAC Iteration 10000
MAC verified OK
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Bag Attributes
    friendlyName: elasticsearch
    localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIMnUYi4qtVnECAggA
- - - - - - - 8< - - - - - - -
     MERCELESSLY  SKIPPED
- - - - - - - >8 - - - - - - -
q4c=
-----END ENCRYPTED PRIVATE KEY-----
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Certificate bag
Bag Attributes
    friendlyName: elasticsearch
    localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
subject=/CN=elasticsearch
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIVAP5QDEzgpciNfp9kgbCxZw0mOKRJMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIxMTIxMzE0MzQzOVoXDTI0MTIxMjE0MzQzOVowGDEWMBQG
A1UEAxMNZWxhc3RpY3NlYXJjaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAJOXlUUKEPuZHsHvzhWy4Q5VwhhBmm8z6ZHq/+x0bajRxO6vxGsrg4TXzhyb
GHvu5yh6uLMqObJOZ4Z0g9dypAA3UgYUXg6l/Li2aQQqOGlkF0awYrWWR6Mn0rb7
1ZgTyrI3jkP3OU8Vz1KRsDWXPzjwmuUvQw4AF4a4eyi+UVqkM+DcjgmKvg+uJQY6
/4Ca5VlDSjJwMauDEq8izHCEsqMJgWfHLJcaAgfyIgnd9pRFtNxps9orHDGzC+1H
U6LDxH9h7F08ngG0POrTMg7B4rZ4lnOuarj0QoPKd3HHIPDzFDikr/2n/+eHhsjB
lATSVAcOsgKnJSPDhDpgvsLh5S0CAwEAAaNnMGUwHQYDVR0OBBYEFHY+4OCA54AC
FjRaYO8hEqQnTBGuMB8GA1UdIwQYMBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMBgG
A1UdEQQRMA+CDWVsYXN0aWNzZWFyY2gwCQYDVR0TBAIwADANBgkqhkiG9w0BAQsF
AAOCAQEAkR9V8QosOBaYAqaesOHbX0ghPvSxCuhxrgHqoAzk3mFIxWUQOWl/zHgo
Af2nnJ6odWQ5rTPMIzUB85vSBj2lZb2LIfp6PL2ZiV7PKOqMAo/0sKz1FliXEoOj
AX4P6MDZEoaNeowHIEG0v/tYHMThqAzOXLfiCYPjnanQYTXe9s1UzS3jWEeowLUu
n3e922Mm32qQtaVE4Py/J2w1jV9o2i8Jlx/FfzUaMmfTCNhMdcioxIiWYOwymcUv
zTU3kxFqym8ojNLmkD3nAgNV34xPHNlABQL3y0x1cXcEiHMnzPgD654wX1/ot0Ul
30kd7eVSCX0gEqqL/9gIICxEPtyreA==
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=/CN=Elastic Certificate Tool Autogenerated CA
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUKIlzkBKxxPYjUmdGDB1aefkP5vMwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMjExMjEzMTQzMTMxWhcNMjQxMjEyMTQzMTMxWjA0MTIwMAYD
VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK39C3JleGydZtMh4kjdH+6y
92yss/ogwsZRpDL8yNp+pzrO8PUc3HhKuaxaaYCYFIokhFS1neRoZeTgvcJ1+IOI
BOGYLe1nJ6xUJhbr75JlkjOq7XNbGiArEoqzvv6WzmybVfxrT+pvFoz1aigIZaqw
6EL7+IcAUdcUuptIpYefnnCkeCW2p7zjSCERTaEJVOiD+gTbi1jVujTLzNyLpug4
SuWa0/olkG0lFpoP3jIoyMlwvorJuRANMaWXA4SH+vn/vmvZeIZE5gm7FMnOrn6l
oiVFr0tbZZgVO3fdW1gTAie3iz2i8o+sV5/U87gsjz2yHUVZNVjV6qt4UzElJEkC
AwEAAaNTMFEwHQYDVR0OBBYEFJ6R4wn2GlPuu5x8b1ghaCENQHzkMB8GA1UdIwQY
MBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAFwMYu4QFzRf17TLYCnIFWf18zBgKS7HtIcc+yeORzgIfbmC
rcKhVzxDNwEy8dfl8D3zk0t/IbOoBFWzKWJjHK5brICoZ13CQ4qdhWE2GBcQbwRL
9CDlbEN520XQk5UMOMaLy6AwPJNkzNLu8p6HEYiupmC0t4NZCSlYWHoqwmhgX7gD
PY+OLEHYpmNRLIukRrw7OpKcC2dMbTPqeYEMXrttuICvEpwWxfJ+hU4sh2vT1dSs
OPVsanS+9DtAZRgDHG0UiPYAXmAh8wYT5DY6wm5jGXa7CXLuvZqLN/PAB+wJvDDy
Mr5Qu7Y+zG9EFb05vNiNme2YU/kbpMRLMW8MEGo=
-----END CERTIFICATE-----

And when all the nodes start, they try to communicate to each other, but they can't do that, because they don't seem to show their certificates to each other.

{"type": "server", "timestamp": "2021-12-13T14:47:38,292Z", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-master-0", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:60682}", "cluster.uuid": "p-UpYVI1QK2dPGN0TMF3Lg", "node.id": "yLOUYcjeS1KIKr6Y6oNhoA" ,
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
...}

Would anyone be so kind as to help me to understand what did I do wrong? Is the certificate inappropriate? Should I have added something to the configuration files?

Lots of thanks in advance.

v_melnyk · December 13, 2021, 6:00pm

Excuse me, I've just realized that a connections from 127.0.0.1:foo to 127.0.0.1:9200 are readiness probes performed by K8S.

Well, that means that I actually do NOT have any serious difficulties with the certificates issued by Elasticsearch-certutil. Though when I try to utilize my own CA's certificate, I see something very similar (the same Empty client certificate chain message), but I clearly see that the connections are initiated by other nodes.

I'll post another topic with all the details a bit later.

system · January 10, 2022, 6:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Ssl between nodes Elasticsearch elastic-stack-security	4	316	December 15, 2023
Elasticsearch https secure communication Elasticsearch elastic-stack-security	4	535	March 11, 2020
Sorting out TLS for ELK, using Helm Chart, in AKS Elasticsearch elastic-stack-security , docker	4	2407	November 22, 2021
TLS transport in single node cluster Elasticsearch	1	219	July 26, 2022
Unable to enable https and internode communication in 2 node cluster Elasticsearch elastic-stack-security	17	1543	June 18, 2020

Can't get nodes communicate to each other via TLS-secured transport

Related topics