# Unbalanced primary shards affects index performance?

**URL:** <https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489>\
**Category:** Elasticsearch\
**Created:** [October 26, 2017, 11:22pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489 "2017-10-26T23:22:56Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [October 26, 2017, 11:22pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/1 "2017-10-26T23:22:56Z")

</div>

In a [previous discussion](https://discuss.elastic.co/t/distributing-primary-shards/67804) there's a claim that we shouldn't worry about balancing primary shards across all nodes because primaries and replicas do almost the same amount of work during indexing.

Currently my primaries are highly unbalanced:

 ![Screenshot from 2017-10-26 18-46-26](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7f81b531a2dc613b1365e2f3ce9cd82cbc73c12.png)

I'm having a problem with my bulk index thread queue filling up and tasks being dropped, and the nodes with the most amount of primaries have the most dropped:

```auto
http es5-client01.c.fp:9200/_cat/thread_pool | grep bulk | grep data | sort
es5-data01 bulk 8 172 23634
es5-data02 bulk 1 0 13082
es5-data03 bulk 1 0 0
es5-data04 bulk 1 0 10812
es5-data05 bulk 0 0 1112
es5-data06 bulk 0 0 2071
es5-data07 bulk 0 0 0

```

The fact that some nodes have a non-empty queue and other nodes are showing no active jobs leads me to think that having such an unbalanced cluster must affect indexing performance. Jobs are waiting in queue on on one node when they could be being processed on another if the primaries were more balanced. Am I wrong here?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [October 28, 2017, 1:35am UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/2 "2017-10-28T01:35:30Z")

</div>

Do you have a lot of `update` operations in these bulk requests?

---

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [November 21, 2017, 3:08pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/3 "2017-11-21T15:08:42Z")

</div>

Sorry for the delay, I didn't get a notification about your response.

On this current job, yes, I am updating documents. I use the elasticsearch-py bulk API with `index` commands. However, the `_id`s in the bulk payload already exist so it results in an update.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [November 21, 2017, 4:02pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/4 "2017-11-21T16:02:42Z")

</div>

If you index them with the `index` command there should be no difference between primary and replica loads. Could you run hot threads on the busy node and on a non-busy node and send both here?

---

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [November 21, 2017, 5:37pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/5 "2017-11-21T17:37:31Z")

</div>

Here are the hot threads for the busy node: [https://pastebin.com/KQr0Ynrc/?e=1](https://pastebin.com/KQr0Ynrc/?e=1)

And here for the non-busy node:

```auto
http https://es5-client01.c.fp:9200/_nodes/es5-data03/hot_threads
HTTP/1.1 200 OK
content-encoding: gzip
content-type: text/plain; charset=UTF-8
transfer-encoding: chunked

::: {es5-data03}{yspPeyimQAOzkKy68w14Gg}{T093J3sxQAyqEQsw3d-ZfQ}{10.208.0.128}{10.208.0.128:9300}{ml.max_open_jobs=10, ml.enabled=true}
   Hot threads at 2017-11-21T17:42:29.669Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
    6.2% (30.9ms out of 500ms) cpu usage by thread 'elasticsearch[es5-data03][refresh][T#4]'
     10/10 snapshots sharing following 27 elements
       sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
       java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
       org.elasticsearch.index.IndexWarmer$FieldDataWarmer.lambda$warmReader$2(IndexWarmer.java:164)
       org.elasticsearch.index.IndexWarmer$FieldDataWarmer$$Lambda$2052/1804043570.awaitTermination(Unknown Source)
       org.elasticsearch.index.IndexWarmer.warm(IndexWarmer.java:84)
       org.elasticsearch.index.IndexService.lambda$createShard$1(IndexService.java:343)
       org.elasticsearch.index.IndexService$$Lambda$2014/193368789.warm(Unknown Source)
       org.elasticsearch.index.engine.InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1434)
       org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:198)
       org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:160)
       org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
       org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
       org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
       org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:910)
       org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632)
       org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690)
       org.elasticsearch.index.IndexService.access$400(IndexService.java:92)
       org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832)
       org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [November 22, 2017, 12:54pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/6 "2017-11-22T12:54:22Z")

</div>

That's strange. Are you using custom routing, search preferences or scan/scroll searches by any chance? Which version of Elasticsearch is this? Is elasticsearch.yml file on the overloaded node different from other nodes?

---

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [November 22, 2017, 1:08pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/7 "2017-11-22T13:08:47Z")

</div>

This is ES v5.5.2

There is an index on this cluster with parent/child documents, though it was only doing incremental indexing (~10 docs/minute). There was a much heavier indexing job going on that didn't have any custom routing.

elasticsearch.yml is the same across data nodes.

Also, I didn't know it at the time, but we've since discovered there were network problems with our cloud provider that our cluster is deployed to when I grabbed these hot\_threads. Once the network issues resolved, the number of docs in the queues went to zero rather quickly.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [November 22, 2017, 1:12pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/8 "2017-11-22T13:12:55Z")

</div>

Could you rerun hot\_thread again and this time could you run them a few times? Let's say 5 times with 40 seconds interval between runs. Just to make sure it wasn't a fluke?

---

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [November 22, 2017, 1:28pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/9 "2017-11-22T13:28:43Z")

</div>

Sure: [https://pastebin.com/nJ8j5C5Y](https://pastebin.com/nJ8j5C5Y)

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [November 22, 2017, 1:45pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/10 "2017-11-22T13:45:09Z")

</div>

Sorry, I meant to run hot threads on the busy node and on some other node at the same time. So, we can see how they compare.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2017, 1:45pm UTC](https://discuss.elastic.co/t/unbalanced-primary-shards-affects-index-performance/105489/11 "2017-12-20T13:45:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.