# Can't get nodes communicate to each other via TLS-secured transport

**URL:** https://discuss.elastic.co/t/cant-get-nodes-communicate-to-each-other-via-tls-secured-transport/291676
**Category:** Elasticsearch
**Created:** [December 13, 2021, 2:53pm UTC](https://discuss.elastic.co/t/cant-get-nodes-communicate-to-each-other-via-tls-secured-transport/291676 "2021-12-13T14:53:29Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![v\_melnyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/v_melnyk/32/98820_2.png) [@v\_melnyk](https://discuss.elastic.co/u/v_melnyk)
#### Post date: [December 13, 2021, 2:53pm UTC](https://discuss.elastic.co/t/cant-get-nodes-communicate-to-each-other-via-tls-secured-transport/291676/1 "2021-12-13T14:53:29Z")

</div>

Hello,

I have a question regarding deploying of a cluster of nodes and stuck with an inter-node communications issue. It seems that my mistake and the solution are pretty obvious, but somewhy I can't grasp what exactly is wrong (honestly speaking, I never deployed Elasticsearch before).

So, I have a K8S cluster (bare metal deployment) and I deploy Elasticsearch with the `elasticsearch` Helm chart that is available at the `https://helm.elastic.co` repository. Actually I have 3 deployments, they're called `elasticsearch-client`, `elasticsearch-data` and `elasticsearch-master`, each of these deployments has its own set of Helm-processed values.

`elasticsearch-master` uses the following values:

```auto
---

clusterName: "elasticsearch-logging"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "false"
  data: "false"
  ml: "false"
  remote_cluster_client: "false"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

```

`elasticsearch-data` uses the following values:

```auto
---

clusterName: "elasticsearch-logging"
nodeGroup: "data"

roles:
  master: "false"
  ingest: "true"
  data: "true"
  ml: "false"
  remote_cluster_client: "false"

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

```

`elasticsearch-client` uses the following values:

```auto
---

clusterName: "elasticsearch-logging"
nodeGroup: "client"

roles:
  master: "false"
  ingest: "false"
  data: "false"
  ml: "false"
  remote_cluster_client: "false"

persistence:
  enabled: false

protocol: https

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.client_authentication: required

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

```

It's kind of a hybrid of these 2 examples:

- [helm-charts/elasticsearch/examples/multi at main · elastic/helm-charts · GitHub](https://github.com/elastic/helm-charts/tree/main/elasticsearch/examples/multi),
- [helm-charts/elasticsearch/examples/security at main · elastic/helm-charts · GitHub](https://github.com/elastic/helm-charts/tree/main/elasticsearch/examples/security).

The keystore (and. at the same time, the truststore) is pretty primitive. I've made it with Elasticsearch-certutil:

- `elasticsearch-certutil ca --out ~/elastic-stack-ca.p12 --pass ''`
- `elasticsearch-certutil cert --name elasticsearch --dns elasticsearch --ca ~/elastic-stack-ca.p12 --pass '' --ca-pass '' --out ~/elastic-certificates.p12`.

Actually I'm trying to issue certificates with my own intermediate CA, but I have the same problem with my own certificate, so I decided to issue certificate with Elasticsearch-certutil just to make sure that I really can do it with the recommended tool. And here's the result that I get:

```auto
MAC Iteration 10000
MAC verified OK
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Bag Attributes
    friendlyName: elasticsearch
    localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIMnUYi4qtVnECAggA
- - - - - - - 8< - - - - - - -
     MERCELESSLY SKIPPED
- - - - - - - >8 - - - - - - -
q4c=
-----END ENCRYPTED PRIVATE KEY-----
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Certificate bag
Bag Attributes
    friendlyName: elasticsearch
    localKeyID: 54 69 6D 65 20 31 36 33 39 34 30 36 30 37 39 35 33 32
subject=/CN=elasticsearch
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIVAP5QDEzgpciNfp9kgbCxZw0mOKRJMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIxMTIxMzE0MzQzOVoXDTI0MTIxMjE0MzQzOVowGDEWMBQG
A1UEAxMNZWxhc3RpY3NlYXJjaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAJOXlUUKEPuZHsHvzhWy4Q5VwhhBmm8z6ZHq/+x0bajRxO6vxGsrg4TXzhyb
GHvu5yh6uLMqObJOZ4Z0g9dypAA3UgYUXg6l/Li2aQQqOGlkF0awYrWWR6Mn0rb7
1ZgTyrI3jkP3OU8Vz1KRsDWXPzjwmuUvQw4AF4a4eyi+UVqkM+DcjgmKvg+uJQY6
/4Ca5VlDSjJwMauDEq8izHCEsqMJgWfHLJcaAgfyIgnd9pRFtNxps9orHDGzC+1H
U6LDxH9h7F08ngG0POrTMg7B4rZ4lnOuarj0QoPKd3HHIPDzFDikr/2n/+eHhsjB
lATSVAcOsgKnJSPDhDpgvsLh5S0CAwEAAaNnMGUwHQYDVR0OBBYEFHY+4OCA54AC
FjRaYO8hEqQnTBGuMB8GA1UdIwQYMBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMBgG
A1UdEQQRMA+CDWVsYXN0aWNzZWFyY2gwCQYDVR0TBAIwADANBgkqhkiG9w0BAQsF
AAOCAQEAkR9V8QosOBaYAqaesOHbX0ghPvSxCuhxrgHqoAzk3mFIxWUQOWl/zHgo
Af2nnJ6odWQ5rTPMIzUB85vSBj2lZb2LIfp6PL2ZiV7PKOqMAo/0sKz1FliXEoOj
AX4P6MDZEoaNeowHIEG0v/tYHMThqAzOXLfiCYPjnanQYTXe9s1UzS3jWEeowLUu
n3e922Mm32qQtaVE4Py/J2w1jV9o2i8Jlx/FfzUaMmfTCNhMdcioxIiWYOwymcUv
zTU3kxFqym8ojNLmkD3nAgNV34xPHNlABQL3y0x1cXcEiHMnzPgD654wX1/ot0Ul
30kd7eVSCX0gEqqL/9gIICxEPtyreA==
-----END CERTIFICATE-----
Certificate bag
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=/CN=Elastic Certificate Tool Autogenerated CA
issuer=/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUKIlzkBKxxPYjUmdGDB1aefkP5vMwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMjExMjEzMTQzMTMxWhcNMjQxMjEyMTQzMTMxWjA0MTIwMAYD
VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK39C3JleGydZtMh4kjdH+6y
92yss/ogwsZRpDL8yNp+pzrO8PUc3HhKuaxaaYCYFIokhFS1neRoZeTgvcJ1+IOI
BOGYLe1nJ6xUJhbr75JlkjOq7XNbGiArEoqzvv6WzmybVfxrT+pvFoz1aigIZaqw
6EL7+IcAUdcUuptIpYefnnCkeCW2p7zjSCERTaEJVOiD+gTbi1jVujTLzNyLpug4
SuWa0/olkG0lFpoP3jIoyMlwvorJuRANMaWXA4SH+vn/vmvZeIZE5gm7FMnOrn6l
oiVFr0tbZZgVO3fdW1gTAie3iz2i8o+sV5/U87gsjz2yHUVZNVjV6qt4UzElJEkC
AwEAAaNTMFEwHQYDVR0OBBYEFJ6R4wn2GlPuu5x8b1ghaCENQHzkMB8GA1UdIwQY
MBaAFJ6R4wn2GlPuu5x8b1ghaCENQHzkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAFwMYu4QFzRf17TLYCnIFWf18zBgKS7HtIcc+yeORzgIfbmC
rcKhVzxDNwEy8dfl8D3zk0t/IbOoBFWzKWJjHK5brICoZ13CQ4qdhWE2GBcQbwRL
9CDlbEN520XQk5UMOMaLy6AwPJNkzNLu8p6HEYiupmC0t4NZCSlYWHoqwmhgX7gD
PY+OLEHYpmNRLIukRrw7OpKcC2dMbTPqeYEMXrttuICvEpwWxfJ+hU4sh2vT1dSs
OPVsanS+9DtAZRgDHG0UiPYAXmAh8wYT5DY6wm5jGXa7CXLuvZqLN/PAB+wJvDDy
Mr5Qu7Y+zG9EFb05vNiNme2YU/kbpMRLMW8MEGo=
-----END CERTIFICATE-----

```

And when all the nodes start, they try to communicate to each other, but they can't do that, because they don't seem to show their certificates to each other. ☹

```auto
{"type": "server", "timestamp": "2021-12-13T14:47:38,292Z", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-master-0", "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:60682}", "cluster.uuid": "p-UpYVI1QK2dPGN0TMF3Lg", "node.id": "yLOUYcjeS1KIKr6Y6oNhoA" ,
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
...}

```

Would anyone be so kind as to help me to understand what did I do wrong? Is the certificate inappropriate? Should I have added something to the configuration files?

Lots of thanks in advance.

---

<div class="post-metadata">

### Author: ![v\_melnyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/v_melnyk/32/98820_2.png) [@v\_melnyk](https://discuss.elastic.co/u/v_melnyk)
#### Post date: [December 13, 2021, 6:00pm UTC](https://discuss.elastic.co/t/cant-get-nodes-communicate-to-each-other-via-tls-secured-transport/291676/2 "2021-12-13T18:00:18Z")

</div>

Excuse me, I've just realized that a connections from `127.0.0.1:foo` to `127.0.0.1:9200` are readiness probes performed by K8S.

Well, that means that I actually do NOT have any serious difficulties with the certificates issued by Elasticsearch-certutil. Though when I try to utilize my own CA's certificate, I see something very similar (the same `Empty client certificate chain` message), but I clearly see that the connections are initiated by other nodes.

I'll post another topic with all the details a bit later.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 10, 2022, 6:00pm UTC](https://discuss.elastic.co/t/cant-get-nodes-communicate-to-each-other-via-tls-secured-transport/291676/3 "2022-01-10T18:00:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.