Reindex from remote source doesn't work with either API key or user/PW?

I am trying to move an index from a SRC_HOST to a TARGET_HOST.
I run this command on TARGET_HOST:

POST _reindex
{
"source": {
"remote": {
"host": "https:SRC_HOST:9200",
"username": "elastic",
"password": "PW"
},
"index": "pages"
},
"dest": {
"index": "pages"
}
}

I've also modified elasticsearch.yml in the TARGET_HOST container to include these params:

reindex.remote.whitelist: ["SRC_HOST:9200"]
reindex.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca/ca.crt"

and copied ca.crt from SRC_HOST to TARGET_HOST and changed its owner to elasticsearch.

but get this error:

{"type": "s_s_l_handshake_exception",
"reason": "(certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}

trying the same command but with "api_key": "WW5SazFaMEJ...=="
instead of user/PW lines, but get the same error.

what am I doing wrong, please?

The certificate_unknown / PKIX path validation failed error is from the TLS handshake, which happens before Elasticsearch sends the username, password, or API key. Both attempts failing the same way is expected: the target node is connecting to the source, and it does not trust the certificate the source presents.

reindex.remote.whitelist is already working. A whitelist rejection is a different error (not whitelisted in reindex.remote.whitelist), and this request got far enough to start TLS.

Trust for remote reindex is configured on the target node, not in the request body. reindex.ssl.certificate_authorities has to point at the CA that signed the certificate the source presents on port 9200. Make sure the file the target is reading is the source's CA and not the target's own. Both settings are static, so the target node has to be restarted after elasticsearch.yml changes.

You can confirm from inside the TARGET container:

curl --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt https://SRC_HOST:9200

If that fails with a certificate error, that file isn't the CA the source uses. If it succeeds, the file is fine, and the next thing to check is that the target node was restarted with the setting pointing at it.

thanks for your help @Elliot_Barlas ! i've confirmed that the certificate mentioned in reindex.ssl.certificate_authorities is the same as that copied from SRC_HOST. but now i am getting a different error when i run the command you suggest within the TARGET_HOST container:

curl --cacert /usr/share/elasticsearch/config/certs/CERT_NAME.crt https://192.168....

curl: (60) SSL: no alternative certificate subject name matches target host name '192.168...'

how do i associate a host name with a certificate?
(i realize these are not really ES questions so I esp. appreciate your help neverthless!)

Hi @rik If you do, curl -v it will show you what you're sending versus what's valid. You may just need to use the host name in the URL, not the IP.

If not, you're going to have to create new certificates with the subject names (hostnames) or IPS that you want to use.
Create them and install them in elasticsearch.
This is pretty common.

But if you look at the curl and see what you have to match with then it'll probably work because it's doing certificate validation which validates the subject names or IPS that must match what you're using in the URL versus what's in the certificate.

If you're in containers probably not going to work the self generate certs probably have container names and are internal IPS, not external IPS or host names

The verbose log does not make it any more clear to me what the issue is.
a snip showing change from IN to OUT

	...
	* TLSv1.3 (IN), TLS handshake, Finished (20):
	{ [52 bytes data]
	* TLSv1.2 (OUT), TLS header, Finished (20):
	} [5 bytes data]
	* 
	...

If not, you're going to have to create new certificates with the subject
names (hostnames) or IPS that you want to use.
Create them and install them in elasticsearch.

how do I do that, please?

Hi @rik
Perhaps If you showed the entire curl -v command and output I might be able to help. It should show what you're sending and what available SAN in IP are available to be matched.

that snippet is not really helpful.

You mentioned containers if you showed a docker compose we might be able to help. We're are trying to help solve this and you're giving tiniest slivers of information.

More information is the quicker way to solution than feeling around the elephant... I understand the trade-offs of security and stuff like that

If you want to anonymize things you can do that just be consistent.

To generate new certs perhaps look at this ..

apologies, I trimmed the log to try to keep its focus; my bad. full log below. I don't see it including "... what available SAN in IP are available to be matched" though? I'll follow the hints toward generating a new cert with SAN.

podman exec ocd4oak-es01-1 curl -v --cacert /usr/share/elasticsearch/config/certs/mjq_ca.crt  https://192.168.0.240:9200 
*   Trying 192.168.0.240:9200...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.0.240 (192.168.0.240) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /usr/share/elasticsearch/config/certs/mjq_ca.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS header, Unknown (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [32 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [850 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS header, Unknown (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=es01
*  start date: Sep  1 23:52:18 2026 GMT
*  expire date: Aug 31 23:52:18 2029 GMT
*  subjectAltName does not match 192.168.0.240
* SSL: no alternative certificate subject name matches target host name '192.168.0.240'
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Unknown (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} [2 bytes data]
curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.0.240'
More details here: 


curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

you said you wanted to see the docker-compose spec too:

es01:
depends_on:
setup:
condition: service_healthy
image: ``docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
- es01_data:/usr/share/elasticsearch/data
- es01_snapshots:/usr/share/elasticsearch/snapshots
- ${ES_CUSTOM_CONFIG_PATH}:/usr/share/elasticsearch/config/elasticsearch.yml:ro
ports:
- ${CONTAINER_ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${ES_CLUSTER_NAME}
- discovery.type=single-node
- ELASTIC_PASSWORD=${ES_PASSWORD}
- path.repo=/usr/share/elasticsearch/snapshots
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${ES_LICENSE}
- xpack.ml.use_auto_machine_memory_percent=true
mem_limit: ${ES_MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt ``https://localhost:9200`` | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120

reading this Update TLS certificates | Elastic Docs now makes me unsure even which of the xpack.security params (http.ssl, transport.ssl) are the issue!?

 - xpack.security.enabled=true
 - xpack.security.http.ssl.enabled=true
 - xpack.security.http.ssl.key=certs/es01/es01.key
 - xpack.security.http.ssl.certificate=certs/es01/es01.crt
 - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
 - xpack.security.transport.ssl.enabled=true
 - xpack.security.transport.ssl.key=certs/es01/es01.key
 - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
 - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
 - xpack.security.transport.ssl.verification_mode=certificate

@rik Excellent, Thanks we can solve this

As I expected

As suspected....
So you are using 192.168.0.240 to connect
But your cert ONLY has subject: CN=es01 which is the internal container name so the certificate will never match. That is the issue.

So you will need to add these external IPs that you are using to access to the certs.

So assuming you are starting from a trimmed down version of this...

You would simply add

          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 192.168.0.240\n"\ <<< THIS and any other IPs 

This will then match and you should be good to go!

  1. Do a compose down (the data volumes should survive the down, but I am not sure exactly how you mounted etc)
  2. Edit your compose
  3. Clean the certs from the volume (You need to find them and clean them out OR just delete the cers volume)
  4. Then compose up which should rebuild the certs with your new IPS and SANs in there

Let me know if that make sense....

ok, I added the line you suggested mentioning 192.168.0.240 to setup service script creating certs es01.ip, and also had to podman volume rm ocd4oak_certs in order to clear the existing certs. but now (instead of getting any of the subject: CN=es01 info you highlighted last time) I get:

SSL certificate problem: unable to get local issuer certificate

full log below:)

(ocd4oak) rik@wayne ocd4oak % podman exec ocd4oak-es01-1 curl -v --cacert /usr/share/elasticsearch/config/certs/ca/ca.crt  https://192.168.0.240:9200

Trying 192.168.0.240:9200...
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.0.240 (192.168.0.240) port 9200 (#0)

ALPN, offering h2

ALPN, offering http/1.1

CAfile: /usr/share/elasticsearch/config/certs/ca/ca.crt

TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]

TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]

TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]

TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]

TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]

TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]

TLSv1.2 (IN), TLS header, Unknown (23):
{ [5 bytes data]

TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [32 bytes data]

TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [850 bytes data]

TLSv1.2 (OUT), TLS header, Unknown (21):
} [5 bytes data]

TLSv1.3 (OUT), TLS alert, unknown CA (560):
} [2 bytes data]

SSL certificate problem: unable to get local issuer certificate
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Again, can you show me the full curl -v and the output every time.

Maybe that is the output of the whole curl? I don't understand why it's pod man. I just wanted you to curl from outside the container to the container like you did above.
Repeating the same commands until we get it fixed

Also please show me your compose

And what commands you ran, the more you show me, the more I can help

Exactly how did you create the certs? I want to see everything
I want to see how you constructed the cert?

Did you use the setup script or what?

Are the permissions correct on the cert files? They need to be readable

More is better every command otherwise I'm just guessing we're doing asynchronous debugging over the Internet. It could not be any more challenging :slight_smile:

I'm not as familiar with podman

Get me all that and I'll take a look a little bit later

I'm even open to trying POD man since I didn't try it before. So give me good stuff and we'll see what we can do

I put an update. I just want the plain old curl from the host to the container like you did before

Did you create a new CA?
If so, you're going to have to put it everywhere and all the certs need to be using that same CA

By the way, this is all normal pain in the PITA cert stuff not really elastic specific.

hi @stephenb , sorry to be frustrating

  • the full curl -v log is above

  • the ES service spec in docker-compose is above:

  • the certs are now (after blowing away the existing certs volume) only those created by the same setup service you pointed; their state (permissions, etc) are just what elasticsearch-certutil builds using the instances.yml built there
  • the only commands i've been using thru this are podman compose up -d and down
  • podman is (almost) plug compatible with docker

(just seeing your update/new message). responding to that next!

I'll build a full working docker compose here in a bit. I've got a few things to do.
I think your ca is out of sync or something
So you're okay cleaning out everything and starting?

And just to be clear by the way, you can turn off certificate validation if you want, but I think you probably want to figure out how to do this right.

I'll give you something working here in a bit. I think you're close. I think you're just out of sync or something

But it would help if you showed me the entire docker compose but I don't see the setup stuff. I don't see the commands that created the certs. That's what's super important

Hi @rik Here you go

FIrst This is because you created a new ca.crt in the docker but are not using it in the curl command I repeated that. I got the same error.

So here is my fully working compose and .env
My local IP is 192.168.86.127

Compose Click to expand
services:
  setup:
    container_name: setup
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "      - orion\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 192.168.86.127\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    container_name: es01 
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true    
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    container_name: kibana 
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
.env Click to expand

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=mypassword

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=mypassword

# Version of Elastic products
#STACK_VERSION=9.3.0
#STACK_VERSION=9.4.1
STACK_VERSION=9.5.4

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
#LICENSE=basic
LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200

# Port to expose Kibana to the host
KIBANA_PORT=5601

KB_ENCRYPTIONKEY=1234-5678-9012-3456-9999-8888-1111-0000

# Increase or decrease based on the available host memory (in bytes)
# 1GB
#MEM_LIMIT=1073741824 
MEM_LIMIT=2173741824 

# For discuss-single
ES_MEM_LIMIT=2173741824 
KB_MEM_LIMIT=2173741824 

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

Curl Command with all the goodies.
I had to clean everything out deleted the volumes maually
I have found you have to really careful cleaning out the volumes and making sure after you create the new CAs certs they that you use the new CAs
A New CA is create so you need to make sure you use that with the curl / clients etc

curl -v --cacert ./ca.crt -u elastic:mypassword https://192.168.86.127:9200
*   Trying 192.168.86.127:9200...
* Connected to 192.168.86.127 (192.168.86.127) port 9200
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: ./ca.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=es01
*  start date: Oct 25 20:57:18 2026 GMT
*  expire date: Oct 24 20:57:18 2029 GMT
*  subjectAltName: host "192.168.86.127" matched cert's IP address!  <<< YAY!
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
* using HTTP/1.x
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: 192.168.86.127:9200
> Authorization: Basic ZWxhc3RpYzpteXBhc3N3b3Jk
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 532
< 
{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "pgF9kcGiRX6xKCFdXstwWQ",
  "version" : {
    "number" : "9.5.4",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "9170df19cae1adb107b7b489b4d82dec66d7a337",
    "build_date" : "2026-09-09T22:42:53.976833287Z",
    "build_snapshot" : false,
    "lucene_version" : "10.5.1",
    "minimum_wire_compatibility_version" : "8.19.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 192.168.86.127 left intact

Hope this helps.
And no frustration at all. Just a process that works for me and