Is reindex from remote included in Python client

Elastic Stack Elasticsearch
1

The Elasticsearch documentation specify the ability to reindex from remote. Can this be done through Python client? I can't find any example. This is what I got which returns error:

host = 'https://XXXXXXXXX'
indexname = 'my_index'
dest_index = 'my dest index'

query = {
"source": {
"remote": {
"host": host
},
"index": indexname
},
"dest": {
"index": dest_index
}
}
result = es.reindex(query, wait_for_completion=True, request_timeout=10000)

elasticsearch.exceptions.RequestError: RequestError(400, 'x_content_parse_exception', '[1:172] [reindex] failed to parse field [source]')

The reindex works without the remote key in the source.

Thanks for your help

2

Hi Shahab, welcome to the community!
The reindex function is indeed implemented for the python client, you can see it in the docs here.where it also mentions this can help you fetch documents from a remote cluster as well.

Does the reindex query you are using work when sending it directly to the dev console? Then we can see if it is a query issue or something specific to the python client implementation.

Just to double check the cluster is whitelisted:

  • If reindexing from a remote cluster, you must explicitly allow the remote host in the reindex.remote.whitelist setting of elasticsearch.yml. See Reindex from remote.

Also, what kind of authentication method are you using for the remote cluster? Can you try adding the username / password in the host part of the request as well?

Hope this helps!
Cheers,
Iulia

3

Hi iulia,
I get the same error when I try it directly in dev tool. I am running some test if the reindex from remote could be used for our production cluster reindexing. In my test the host is the same as my destination cluster. I tried the username/password but get the same error message.
Thanks

Shahab

4

I think in this case snapshot and restore (instead of reindex) would be better fitted. It would be faster in terms of performance, not to mention have the extra benefit of the backup feature, especially if you're looking at this for a production environment.

Have you looked into this option as well? See docs here.

You can implement this in python as well - see docs

And this answer is a bit old but can be a good start for the format.

5

Thank lulia, much appreciate your help.
Would snapshot restore upgrade the version of the indexes?
Some of our indexes are version 6.7 and we need them to be upgraded hence we are looking at the re indexing .
Thanks
Shahab

6

Depends on the versions you're upgrading from and to, we have a table showing the compatibilities here:

You cannot use the snapshot to restore back to an earlier version, only moving forward.
From 6.7 you should be able to use this method to upgrade to 6.8
7.0–7.1, 7.2–7.17. or 8.3-8.10.

7

Many thanks iulia. I will try this method.

Shahab

8

you're welcome! hope it works, otherwise you'll find us here :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.