Reindex not copying every document that exists on the source index

Elastic Stack Elasticsearch
1

I have around 20 indexes to reindex after some mapping changes but on any index that has a high number of documents e.g 500.000 the reindexing process stops abruptly and only a random number of document is reindexed.

Let's take for example this one : s3-data-3006 is the source index and s3-data is the destination.

Doing this :

    POST _reindex?
     {
       
       "conflicts": "proceed",
       "source": {
         "index": "s3-data_3006"
       },
       "dest": {
         "index": "s3-data",
         "op_type": "create"
       }
    }

Only copies around 150k documents, here's the results :

 {
   "took" : 48644,
   "timed_out" : false,
   "total" : 152189,
   "updated" : 0,
   "created" : 0,
   "deleted" : 0,
   "batches" : 153,
   "version_conflicts" : 152189,
   "noops" : 0,
   "retries" : {
     "bulk" : 0,
     "search" : 0
   },
   "throttled_millis" : 0,
   "requests_per_second" : -1.0,
   "throttled_until_millis" : 0,
   "failures" : [ ]
 }

If trying to compare the count or size here's what I get :

    health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   s3-data      ucG_JHb6QVGuUoSqBupCiQ   5   1     152189        44553    725.7mb        362.8mb
    green  open   s3-data_3006 ig0sA_gAS_iQdXVTJ6N2fw   5   1    2461654       721888      1.2gb        619.5mb

And this is everytime I try to reindex this or any particular index, it just stops abruptly everytime with nothing returned. I'm trying to understand why would this happen. This is one of the smaller indexes with around 1.2GB of data, if I can't reindex this one than imagine trying to reindex one of the indexes that contains 35GB of data.

Please let me know if you need any further information

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.