PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Elastic Stack Elasticsearch
1

Scenario:

  • Two single-node clusters (semi-production)
  • Remote cluster configured successfully on both

Cluster Migration

cluster.name: sitelight
node.name: node-0
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: SOME.IP
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  truststore.path: certs/elastic-certificates.p12
cluster.initial_master_nodes: ["elk.domain.tld"]
http.host: 0.0.0.0
path.repo: /mnt/nfs/elasticsearch/cluster01,/mnt/nfs/elasticsearch/cluster02
reindex.remote.whitelist: ["REMOTE.IP:9300", "elk0.domain.tld:9300"]

{
  "elk-cluster": {
    "connected": true,
    "mode": "sniff",
    "seeds": [
      "elk0.PRODUCCTION:9300"
    ],
    "num_nodes_connected": 1,
    "max_connections_per_cluster": 1,
    "initial_connect_timeout": "30s",
    "skip_unavailable": false
  }
}

Production Node

cluster.name: elk-cluster
node.name: node-0
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: SOME.IP
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  truststore.path: certs/elastic-certificates.p12
cluster.initial_master_nodes: ["elk0.domain.tld"]
http.host: 0.0.0.0
path.repo: /mnt/nfs/elasticsearch/cluster01
reindex.remote.whitelist: ["REMOTE.IP:9300","elk.domain.tld:9300"]


{
  "sitelight": {
    "connected": true,
    "mode": "sniff",
    "seeds": [
      "elk.SERVER_MIGRATION:9300"
    ],
    "num_nodes_connected": 1,
    "max_connections_per_cluster": 1,
    "initial_connect_timeout": "30s",
    "skip_unavailable": false
  }
}

Certificates were generated with certutil, using a password:

elasticsearch-certutil ca
elasticsearch-certutil cert --ca elastic-stack-ca.p12

In both nodes the passwords were added to the keystore

❯ elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
Enter value for xpack.security.transport.ssl.keystore.secure_password:
❯ elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
Enter value for xpack.security.transport.ssl.truststore.secure_password:
❯ elasticsearch-keystore list
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

scp -p elk.domain.tld:/usr/share/elasticsearch/elastic-stack-ca.p12 /usr/share/elasticsearch/
scp -p elk.domain.tld:/etc/elasticsearch/certs/elastic-certificates.p12 /etc/elasticsearch/certs/

The permissions were adjusted because the creation tool was giving incorrect permissions such as user and group.

Now, when I try to remap on the remote, using

POST _reindex?pretty
{
  "source": {
    "index": "modems-2024061201",  
    "remote": {
      "host": "https://elk..domain.tld:9300",
      "username": "elastic",
      "password": "PaSWoRd_Elastic_user"   
    },
    "query": {
      "match_all": {}  
    }
  },
  "dest": {
    "index": "modems-2024101801" 
  }
}

Add get

{
  "error": {
    "root_cause": [
      {
        "type": "s_s_l_handshake_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
      }
    ],
    "type": "s_s_l_handshake_exception",
    "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
    "caused_by": {
      "type": "validator_exception",
      "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
      "caused_by": {
        "type": "sun_cert_path_builder_exception",
        "reason": "unable to find valid certification path to requested target"
      }
    }
  },
  "status": 500
}

I don't understand what I did wrong following the doc