We are trying to reindex 2+Tb indexed data between 2 clusters:
Cluster size: 6
Reindexing commands:
curl -XPOST "http://localhost:9200/_reindex?scroll=5m&wait_for_completion=false" -H 'Content-Type: application/json' -d'
{
"source": {
"remote": {
"host": "http://elasticsearch-xyz-cluster-master:9200",
"socket_timeout": "5m",
"connect_timeout": "5m"
},
"index": "xyz_index"
},
"dest": {
"index": "xyz_index_2"
},
"conflicts": "proceed"
}'
We are noticing some blocks here. Reindexing is failing at some level. Can we adjust some cluster settings to accomplish this?
[2021-02-24T17:40:52,810][INFO ][o.e.t.LoggingTaskListener] [elasticsearch-master-0] 5328 finished with response BulkByScrollResponse[took=6.1h,timed_out=false,sliceId=null,updated=0,created=178506000,deleted=0,batches=178506,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[{"shard":-1,"status":429,"reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@588a1b02 on QueueResizingEsThreadPoolExecutor[name = elasticsearch-master-4/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 512.4ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@78c3882f[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 4424686]]"}}, {"shard":-1,"status":429,"reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@5ff927f7 on QueueResizingEsThreadPoolExecutor[name = elasticsearch-master-4/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 512.4ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@78c3882f[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 4424686]]"}}, {"shard":-1,"status":429,"reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1a3609dd on QueueResizingEsThreadPoolExecutor[name = elasticsearch-master-4/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 512.4ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@78c3882f[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 4424686]]"}}]]
I gave try with increasing the CPU count from 8-16, but still no use.