ES Cross Cluster Replication Issue

[sidsniper (sid)]

Hi,

I am getting below error while trying to setup ccr:

"fatal_exception" : {
"type" : "es_rejected_execution_exception",
"reason" : "rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1e0fb72 on QueueResizingEsThreadPoolExecutor[name = localhost/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 54.1ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@4164846[Running, pool size = 7, active threads = 7, queued tasks = 1090, completed tasks = 1734719]]"

Can someone please help me since I am new with this.

ES Version: 6.8

Thanks,
Sid

[jasontedor]

This means that the search queue on your remote (where the follower shard is pulling from) is at capacity, and rejected the search requests the follower runs on the remote cluster to find the changes that it needs to replicate. This means that your remote cluster is overloaded.

[jasontedor]

I opened https://github.com/elastic/elasticsearch/pull/49213 so that this is no longer considered fatal.

[sidsniper (sid)]

Thanks @jasontedor. I wanted to understand one more thing. Currently we have an ELB in front of Elastic Data nodes (ELB:9200). For replication, we need to apply cluster settings something like below:
curl -X PUT "10.167.65.123:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"persistent" : {
"cluster" : {
"remote" : {
"us_east_cluster" : {
"seeds" : [
"10.24.21.168:9300"
]
}
}
}
}
}
'

So can we use ELB:9300 for seeds instead of node IP's since they are not reliable? Currently I am not sure, if the replication will break if the node IP's get refreshed within the cluster.

Thanks,
Sid

[sidsniper (sid)]

@jasontedor can you please help here with my above query?

[jasontedor]

Why are your node IP's changing? Can you refer to them consistently by name?