# Index speed is greatly reduced when doing replication **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2019, 4:12am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667 "2019-03-25T04:12:36Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\ **Post date:** [March 25, 2019, 4:12am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/1 "2019-03-25T04:12:36Z")
hi, guys We use ES for log management and ES cluster is built by hot-cold architecture. One physical host holds one hot node and one cold node. Hot node and cold node share Cpu, memory, but use different storage( ssd for hot node, sata for cold node). Logs of today are bulked to hot node and replica of index is set to 0. Logs of yesterday were moved to cold node. After the move was finished, the replica of index was set to 1. But we found that when doing replication on cold node, the index speed is greatly reduced. Load of physical host is not high and We checked the thread pool and found that lots of bulk request were rejected. When index speed was impacted, we set `cluster.routing.allocation.node_concurrent_recoveries` to 0, the index speed would recover. Any suggestions? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 25, 2019, 6:11am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/2 "2019-03-25T06:11:49Z")
Which version of Elasticsearch are you using? What is the specification of the hosts with respect to CPU, RAM and network? How large are your shards? How many nodes do you have in the cluster? ---
**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\ **Post date:** [March 25, 2019, 9:04am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/3 "2019-03-25T09:04:01Z")
Es version: 5.4.3 CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz Memory: 256GB network: 10GB\*2 disk: 1.6T ssd \* 4, 8T \* 12 sata 43 nodes in cluster: 20 hot, 20 cold, 3 master logs index daily: 30TB/day ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 25, 2019, 9:51am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/4 "2019-03-25T09:51:41Z")
The replication of yesterday's index shouldn't interact at all with today's indexing _within Elasticsearch_ since they're on separate nodes. I therefore think that this interaction is via some other channel. The nodes share CPU and memory, and also both IO and network bandwidth, as well as other resources like filesystem cache. I think that at least one of these is suffering from contention. I think I would start by looking harder for IO contention. Even though the nodes don't share storage they will be sharing other parts of their IO subsystems. You say you reduced `cluster.routing.allocation.node_concurrent_recoveries` to `0`. What was it set to previously? Do you still see problems if it's set to `1`? Also, you can control the rate of recovery with `indices.recovery.max_bytes_per_sec`. Have you set this high enough that it could be slowing down indexing? If you set it lower does it help? ---
**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\ **Post date:** [March 25, 2019, 11:05am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/5 "2019-03-25T11:05:30Z")
thanks for your reply my es settings: cluster.routing.allocation.node\_concurrent\_recoveries: 1 indices.recovery.max\_bytes\_per\_sec: 400mb So these settings are already at low level. I am wandering "cold node export http port so it also serve bulk request, so will bulk request through cold node be affected when cold node doing replication"? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 25, 2019, 11:37am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/6 "2019-03-25T11:37:10Z")
> [@shjdwxy](#): > > I am wandering "cold node export http port so it also serve bulk request, so will bulk request through cold node be affected when cold node doing replication"? I think you should avoid the cold nodes when performing your bulk indexing in order to be sure there's no interaction. However I don't immediately see how this could be causing the effect you describe. Are the bulk rejections happening on the cold nodes or on the hot nodes? > [@shjdwxy](#): > > indices.recovery.max\_bytes\_per\_sec: 400mb 400MB/s seems quite high - that's over 3Gbps, which could well be enough to saturate your IO bandwidth. Try reducing this to see if that helps. ---
**Author:** ![shjdwxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shjdwxy/32/43102_2.png) [@shjdwxy](https://discuss.elastic.co/u/shjdwxy)\ **Post date:** [March 26, 2019, 2:21am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/7 "2019-03-26T02:21:00Z")
thanks I will try to reduce `indices.recovery.max_bytes_per_sec` to see if this helps ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 23, 2019, 2:21am UTC](https://discuss.elastic.co/t/index-speed-is-greatly-reduced-when-doing-replication/173667/8 "2019-04-23T02:21:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.