Hello,
Recently we observed that bulk indexing was rejected from ES when we were adding another set of replica to one of our index. The index already had one replica and we were increasing the replica to 2. The cluster went Yellow as expected, and the cpu was bit high ~70% during this time than usual (~40%). Below is the error thrown by ES captured at client side of the application. Makes me wonder how does adding a replica affects Indexing performance so much? Is there a better way of dealing with this in future other than increasing number of retries at client side?
RemoteTransportException[[es-286][X.X.X.X:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[es-286]hostname/X.X.X.X:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@596a6c2d on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6499a8d3[Running, pool size = 8, active threads = 8, queued tasks = 55, completed tasks = 119162667]]]