Reindex API not distributing load when using slicing

I have an index with 3 nodes running, each node having 1 primary and 1 replica shard:

index        shard prirep state   node
source-index 0     p      STARTED eck-elasticsearch-es-default-2
source-index 0     r      STARTED eck-elasticsearch-es-default-1
source-index 1     p      STARTED eck-elasticsearch-es-default-0
source-index 1     r      STARTED eck-elasticsearch-es-default-1
source-index 2     p      STARTED eck-elasticsearch-es-default-0
source-index 2     r      STARTED eck-elasticsearch-es-default-2

I have a need to use the reindex API to remove some fields from the existing documents. I create the new index with some recommended settings to increase indexing speed, as such:

put /destination-index
{
  "mappings": {
    "dynamic": "strict",
    "properties": {...}
  },
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "refresh_interval": -1
  }
}

I run the reindex request as follows:

POST _reindex?wait_for_completion=false&slices=auto
{
  "source": {
    "index": "source-index",
    "_source": "${varList}"
  },
  "dest": {
    "index": "destination-index"
  }
}

When I run this request, it creates 3 slices to run in parallel. This follows with the docs from Automatic Slicing that states: “This setting (slices=auto) will use one slice per shard, up to a certain limit”. Given the 3 primary shards as shown above, this all makes complete sense.

What doesn't make sense is that I check the child tasks of the main reindex action, and each child task with the action indices:data/write/reindex is using the same node:

action                     parent_task_id                   type      running_time node
indices:data/write/reindex l7bY43OiRGquG6KcBAa4Jw:327586044 transport 15.3m        eck-elasticsearch-es-default-0
indices:data/write/reindex l7bY43OiRGquG6KcBAa4Jw:327586044 transport 15.3m        eck-elasticsearch-es-default-0
indices:data/write/reindex l7bY43OiRGquG6KcBAa4Jw:327586044 transport 15.3m        eck-elasticsearch-es-default-0

It feels like this is unexpected given the wording of the documentation for using slices with _reindex. What I would expect is that the reindex action would utilize all 3 nodes, each utilizing a thread per primary shard, to write to another primary shard in the destination index. Something like:

source primary shard 1 on node A reindexing to destination primary shard 1 on node A
source primary shard 2 on node B reindexing to destination primary shard 2 on node B
etc...

I bring this up because the reindexing process feels slow and unpredictable with each test run that I do. Just curious about utilizing one node for all 3 slices vs distributing the reindexing to every node and spreading the load. Having all 3 slices being processed by the same node at the same time is much slower than a reindex process with the same number of documents without slices.

Any information on this topic would be helpful. Thanks!

Still curious about this!

Still curious!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.