Re-index operation fails due to search context no longer found

[viniciof (Vinicio Flores)]

Hello everyone,

I'm executing the following reindex task from a 4.1TB~ source index (quite big)

POST _reindex?wait_for_completion=false&slices=20
{
  "source": {
    "index": "puma.compilation.pipeline.96f19f5b-bc84-4d4b-8694-b80a293e78e4-latest",
    "size": 1000,
    "query": {
"range": {
      "ibi_logtime": {
        "gte": "now-9M/M"
      }
    }
    }
  },
  "dest": {
    "index": "puma.compilation.pipeline.96f19f5b-bc84-4d4b-8694-b80a293e78e4-optimized"
  }
}

When it has already copied around 2TB~ the task fails with the following errors:

{
        "shard": -1,
        "reason": {
          "type": "search_context_missing_exception",
          "reason": "No search context found for id [2292522]"
}

This target index has zero replicas, refresh interval is disabled and is composed of 90 shards.

The source index is 10 primary shards only, with 1 replica per shard and refresh interval temporarily disabled for better reindex' performance.

In addition, this is the monitoring metrics of the target and source indexes respectively, if it helps in anything.

Target Index

Source Index

Do you know how to debug and fix this kind of error ? (search context not found )