# Elasticsearch not reassigning shards after node rejoined

**URL:** <https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424>\
**Category:** Elasticsearch\
**Created:** [April 25, 2019, 10:12am UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424 "2019-04-25T10:12:21Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 30, 2019, 7:43am UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424/21 "2019-04-30T07:43:08Z")

</div>

Large number of indices and shards lead to a larger cluster state and more entities to gather stats for/from. David is probably in a better position to quantify potential impact.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 30, 2019, 9:39am UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424/22 "2019-04-30T09:39:18Z")

</div>

> [@rsyne](#):
>
> The question is whether the effect of that would be correlated to the problems I'm experiencing.

I could believe it, yes, some stats are collected per-shard so with thousands of shards it could be slow to obtain those stats. I'd missed the shard count in this cluster but @Christian_Dahlqvist is right it's pretty high and likely to cause issues like this.

> [@rsyne](#):
>
> is it possible (and wise) to simply increase their count in order to make things a little less congested?

Not really. If you're going to run out of management threads then adding more threads will just delay the inevitable. I think it'd be better to increase the timeouts for that stats collection to a level where it reliably completes on your cluster, because this'll limit how many of these requests might be happening in parallel. You only get parallel requests when one times out. Try these:

```auto
xpack.monitoring.collection.node.stats.timeout: 60s
xpack.monitoring.collection.index.stats.timeout: 60s

```

---

<div class="post-metadata">

**Author:** ![rsyne](https://avatars.discourse-cdn.com/v4/letter/r/d78d45/32.png) [@rsyne](https://discuss.elastic.co/u/rsyne)\
**Post date:** [April 30, 2019, 1:10pm UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424/23 "2019-04-30T13:10:35Z")

</div>

Right. I will look into that and will get back to you in a couple of days. Thanks!

---

<div class="post-metadata">

**Author:** ![rsyne](https://avatars.discourse-cdn.com/v4/letter/r/d78d45/32.png) [@rsyne](https://discuss.elastic.co/u/rsyne)\
**Post date:** [May 14, 2019, 9:14am UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424/24 "2019-05-14T09:14:46Z")

</div>

All right, seems that the disk / performance problems were related not to shard count, but to cloud-provider having messed up the VM disks' filesystems.  
Changes were introduced and so far the cluster works fine. I believe the problem to be resolved. Many thanks for your help, guys!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2019, 9:14am UTC](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424/25 "2019-06-11T09:14:48Z")

</div>

Closed on June 11, 2019, 9:14am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/elasticsearch-not-reassigning-shards-after-node-rejoined/178424.md?page=1)