# Reindexing from Remote Cluster Fails

**URL:** <https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219>\
**Category:** Elasticsearch\
**Created:** [October 3, 2019, 5:47pm UTC](https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219 "2019-10-03T17:47:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes)\
**Post date:** [October 3, 2019, 5:47pm UTC](https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219/1 "2019-10-03T17:47:22Z")

</div>

I have two clusters running Elasticsearch & Kibana 7.3. I've been able to reindex documents to the same cluster without issue in the past, however when attempting to [Reindex from Remote](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#reindex-from-remote), it's been reindexing much more slowly and consistently produces errors related to search query:

```auto
    "failures" : [
      {
        "shard" : -1,
        "reason" : {
          "type" : "es_rejected_execution_exception",
          "reason" : "rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@5ef3f878 on QueueResizingEsThreadPoolExecutor[name = remote-data-node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 40.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@445cb597[Running, pool size = 49, active threads = 49, queued tasks = 1021, completed tasks = 2445041]]"
        }
      },
...

```

The remote cluster is about 70ms away, and currently not being searched at all, aside from this.  
The reindex is not throttled: `"requests_per_second" : -1.0`, the entire reindex request is:

```auto
POST _reindex?wait_for_completion=false
{
  "source": {
    "remote": {
      "host": "http://remote-coordinating-node:9200"
    },
    "index": "flow-*",
    "query": {
      "bool": {
        "must": [
          {
            "range": {
              "@timestamp": {
                "format": "strict_date_optional_time",
                "gte": "2019-09-30T23:00:00.000Z",
                "lte": "2019-10-01T23:00:00.000Z"
              }
            }
          }
        ],
        "filter": [
          {
            "match_all": {}
          }
        ],
        "should": [],
        "must_not": []
      }
    }
  },
  "dest": {
    "index": "flow_remote_old"
  }
}

```

Do I need to just implement throttling? I imagine it's this (seemingly lightweight) query that's killing my searches:

```auto
            "range": {
              "@timestamp": {
                "format": "strict_date_optional_time",
                "gte": "2019-09-30T23:00:00.000Z",
                "lte": "2019-10-01T23:00:00.000Z"
              }
            }

```

Edit: this error still occurs even without the query - e.g. attempting to copy one index over:

```auto
POST _reindex?wait_for_completion=false
{
  "source": {
    "remote": {
      "host": "http://remote-coordinating-node:9200"
    },
    "index": "flow-prod-000045",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "match_all": {}
          }
        ],
        "should": [],
        "must_not": []
      }
    }
  },
  "dest": {
    "index": "flow_remote_old"
  }
}

```

Any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes)\
**Post date:** [October 4, 2019, 7:08pm UTC](https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219/2 "2019-10-04T19:08:19Z")

</div>

I've tweaked a couple settings yet still am getting timeouts consistently. I currently can't reindex any docs from a remote cluster.

```auto
POST _reindex?wait_for_completion=false&requests_per_second=500
{
  "conflicts": "proceed",
  "source": {
    "size": 100,
    "remote": {
      "host": "http://remote-coordinating-node:9400"
    },
    "index": "flow-prod-000043",
    "query": {
      "bool": {
        "must": [],
        "filter": [
          {
            "match_all": {}
          }
        ],
        "should": [],
        "must_not": []
      }
    }
  },
  "dest": {
    "index": "flow_remote_old",
    "op_type": "create"
  }
}

```

---

<div class="post-metadata">

**Author:** ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes)\
**Post date:** [October 18, 2019, 11:41pm UTC](https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219/3 "2019-10-18T23:41:08Z")

</div>

Found the error. Some go code would run on a cron every hour to refresh some logstash dictionaries. That code was not thread-limited and would create a spike of ~3k searches per second. This of course, as the error says, would stop our reindexing. Reviewing the Search Rate was a dead-giveaway that something funky was going on.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/9/29f3b7ab2b7d8a807e2a50856afb9c437f01a20e.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2019, 11:52pm UTC](https://discuss.elastic.co/t/reindexing-from-remote-cluster-fails/202219/4 "2019-11-15T23:52:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.