Elasticsearch reindex API is not able to copy the documents

Elastic Stack Elasticsearch
#1

Hi,
I am using Elasticsearch 7.8.0. I am trying to copy documents from one index to another using reindex API. I tried 2 cases.
Case 1. Only source index is present in the Elasticsearch cluster. Destination index is not pre-created and it got created by reindex API. In this case the documents got successfully copied into destination index.

Case 2. Both source and destination index(pre-created) are present in ES cluster. Now when I run reindex API, even-though the reindex ran without any errors, the documents did not get copied into destination index. After 5-10 minutes all the documents got copied into destination index. I have tried the same command with wait_for_completion=true. But the curl command is coming out and it is not waiting for the process to get completed. I can see that the destination index still has document count 0. After 5-10 minutes the document count got increased.

Note - Source Index size is in kb and it has less documents. But still it is taking 5-10 minutes to copy when I try reindex API as mentioned in Case 2. If I try reindex as mentioned in Case 1 immediately documents got copied into destination index.

Below is the command tried and its output-

curl -X POST "ES-IP:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "index-1
},
"dest": {
"index": "index-2",
}
}
'
{"took":224,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":}

Can you please tell me why is the Case 2 not working as expected?

#2

Is that the exact command you are running?
I only ask as you are missing ", from the end of index-1.

#3

Hi,
Sorry. There was a typo while pasting above and I have tried the right command in my lab. Below is the exact command.

curl -X POST "ES-IP:9200/_reindex?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "my-index-1"
  },
  "dest": {
    "index": "my-new-index-1"
  }
}
'

Got successful output response also-

{"took":224,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":}

But destination index doesn't have any document count. I can see after 5-10 minutes document count got increased.(I have tried Case 2 as mentioned in above post).
Am I missing something here? Please guide me.

#4

How many in the source index?

#5

Hi,
The source index has document count 27.

#6

Then it looks like the reindex has worked, based on the API response.

What is the output from the _cat/indices/my-*?v API?

#7

Hi,
Here the destination index is pre-created.

[es ~]$ echo $source_index
my-index-1
[es ~]$ echo $dest_index
my-new-index-1
[es ~]$ curl -X POST  -k --cert /etc/elasticsearch/certs/clientCrt --key /etc/elasticsearch/certs/clientKey https://elasticsearch:9200/_reindex?wait_for_completion=true -H 'Content-Type: application/json' -d'{ "source": {"index": "'$source_index'" }, "dest": {"index": "'$dest_index'"}}'
{"took":190,"timed_out":false,"total":27,"updated":0,"created":27,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}
[es ~]$  curl -k --cert /etc/elasticsearch/certs/clientCrt --key /etc/elasticsearch/certs/clientKey https://elasticsearch:9200/_cat/indices/my-*?v
health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   my-index-1     KlaxbHBLRZ24PZfsWhp_JQ   1   1         27            0     63.9kb         23.7kb
green  open   my-new-index-1 kgFum05hQvOD5Rds4--m8g   1   1          0            0       416b           208b

Curl response after 5-10 minutes of reindex API.

health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   my-index-1     KlaxbHBLRZ24PZfsWhp_JQ   1   1         27            0     64.1kb         23.8kb
green  open   my-new-index-1 kgFum05hQvOD5Rds4--m8g   1   1         27            0     47.7kb         23.8kb
#8

Hi,
I can see that the contents of the index got merged as soon as the curl completes. But it is taking time for updating the document count.
Thank you