# Uneven load distribution for search **URL:** \ **Category:** Elasticsearch\ **Created:** [August 16, 2018, 3:25pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742 "2018-08-16T15:25:36Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![stimur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stimur/32/34524_2.png) [@stimur](https://discuss.elastic.co/u/stimur)\ **Post date:** [August 16, 2018, 3:25pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/1 "2018-08-16T15:25:37Z")
Hello! I have weird issue after migrating from 2.4 to 6.3. My cluster consists of 10 nodes: 3 master nodes, (m3.medium, 1CPU/4GB RAM) 3 client nodes running ES, Logstash and Kibana. (c5.xlarge 4xCPU/8GB RAM) 4 data nodes (r4.2xlarge, 61 GB RAM/8 CPU) Data inflow looks like this: - Load balancer sends data to logstash TCP listener - Kibana and Logstash sends data and queries to localhost client node, which then supposed to load balance them to data nodes. The problem here is that I see queries take much longer than they used to in 2.4 and Datadog reports that one of the nodes gets more queries than others ![Selection_063](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66610f47f788645dc6153d614dd179976a358569.png) I don't understand why...Can anybody help here? I'm continue to check for any differences in configuration ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2018, 4:40pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/2 "2018-08-16T16:40:59Z")
Any hotspots due to uneven shard distribution? ---
**Author:** ![stimur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stimur/32/34524_2.png) [@stimur](https://discuss.elastic.co/u/stimur)\ **Post date:** [August 16, 2018, 4:43pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/3 "2018-08-16T16:43:45Z")
I've checked, shards are distributed evenly across all data nodes, I recently switched from 2 shards per index to 4, about a week ago. but timeouts are happening even on shorter periods of time ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2018, 5:48pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/4 "2018-08-16T17:48:52Z")
Do you use routing or any other features that can cause notspots? ---
**Author:** ![stimur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stimur/32/34524_2.png) [@stimur](https://discuss.elastic.co/u/stimur)\ **Post date:** [August 16, 2018, 5:53pm UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/5 "2018-08-16T17:53:22Z")
I'm using routing awareness, based on availability zone, and I just realized that node-0 and node-3 are in the same AZ, while others are in their own... I will re-deploy new instance in a 4th zone and see if that helps ---
**Author:** ![stimur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stimur/32/34524_2.png) [@stimur](https://discuss.elastic.co/u/stimur)\ **Post date:** [August 17, 2018, 2:59am UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/6 "2018-08-17T02:59:31Z")
and now I'm stuck with shard re-balancing 🙂 > [@Speed up shard rebalance](https://discuss.elastic.co/t/speed-up-shard-rebalance/144803): > > I couldn't find anything recent on this topic. I have 4 data nodes, excluded one from routing PUT \_cluster/settings { "transient" :{ "cluster.routing.allocation.exclude.\_ip" : "10.32.130.87" } } and added 5th one to compensate for missing storage, while I replacing excluded node. Now I have 5 servers, 8CPU each, 64 GB of RAM and 10GBps network interfaces (seating in AWS) barely loaded: I/O wait is about 2-5 % per core CPU is only 100% (out of 800) Network and disks are doing no… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2018, 3:08am UTC](https://discuss.elastic.co/t/uneven-load-distribution-for-search/144742/7 "2018-09-14T03:08:51Z")
Closed on September 14, 2018, 3:08am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.