Elasticsearch reindex API is not able to copy the documents

Hi,
I am using Elasticsearch 7.8.0. I am trying to copy documents from one index to another using reindex API. I tried 2 cases.
Case 1. Only source index is present in the Elasticsearch cluster. Destination index is not pre-created and it got created by reindex API. In this case the documents got successfully copied into destination index.

Case 2. Both source and destination index(pre-created) are present in ES cluster. Now when I run reindex API, even-though the reindex ran without any errors, the documents did not get copied into destination index. After 5-10 minutes all the documents got copied into destination index. I have tried the same command with wait_for_completion=true. But the curl command is coming out and it is not waiting for the process to get completed. I can see that the destination index still has document count 0. After 5-10 minutes the document count got increased.

Note - Source Index size is in kb and it has less documents. But still it is taking 5-10 minutes to copy when I try reindex API as mentioned in Case 2. If I try reindex as mentioned in Case 1 immediately documents got copied into destination index.

Below is the command tried and its output-

curl -X POST "ES-IP:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "index-1
},
"dest": {
"index": "index-2",
}
}
'
{"took":224,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":}

Can you please tell me why is the Case 2 not working as expected?

Is that the exact command you are running?
I only ask as you are missing ", from the end of index-1.

Hi,
Sorry. There was a typo while pasting above and I have tried the right command in my lab. Below is the exact command.

curl -X POST "ES-IP:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "my-index-1"
  },
  "dest": {
    "index": "my-new-index-1"
  }
}
'

Got successful output response also-

{"took":224,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":}

But destination index doesn't have any document count. I can see after 5-10 minutes document count got increased.(I have tried Case 2 as mentioned in above post).
Am I missing something here? Please guide me.

How many in the source index?

Hi,
The source index has document count 27.

Then it looks like the reindex has worked, based on the API response.

What is the output from the _cat/indices/my-*?v API?

Hi,
Here the destination index is pre-created.

[es ~]$ echo $source_index
my-index-1
[es ~]$ echo $dest_index
my-new-index-1
[es ~]$ curl -X POST  -k --cert /etc/elasticsearch/certs/clientCrt --key /etc/elasticsearch/certs/clientKey https://elasticsearch:9200/_reindex?wait_for_completion=true -H 'Content-Type: application/json' -d'{ "source": {"index": "'$source_index'" }, "dest": {"index": "'$dest_index'"}}'
{"took":190,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}
[es ~]$  curl -k --cert /etc/elasticsearch/certs/clientCrt --key /etc/elasticsearch/certs/clientKey https://elasticsearch:9200/_cat/indices/my-*?v
health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   my-index-1     KlaxbHBLRZ24PZfsWhp_JQ   1   1         27            0     63.9kb         23.7kb
green  open   my-new-index-1 kgFum05hQvOD5Rds4--m8g   1   1          0            0       416b           208b

Curl response after 5-10 minutes of reindex API.

health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   my-index-1     KlaxbHBLRZ24PZfsWhp_JQ   1   1         27            0     64.1kb         23.8kb
green  open   my-new-index-1 kgFum05hQvOD5Rds4--m8g   1   1         27            0     47.7kb         23.8kb

Hi,
I can see that the contents of the index got merged as soon as the curl completes. But it is taking time for updating the document count.
Thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.