# Daily shard reallocation **URL:** \ **Category:** Elasticsearch\ **Created:** [May 19, 2020, 3:37pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360 "2020-05-19T15:37:38Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 19, 2020, 3:37pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/1 "2020-05-19T15:37:38Z")
in my cluster even when I don't make any changes (zero new index introduce or deleted) daily I have 20-30 shard gets reallocating. why is that? only process going on in cluster is regular ingestion of data. using version 7.6.2 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 19, 2020, 4:06pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/2 "2020-05-19T16:06:06Z")
Hard to say for sure, you've not shared much information here. Two obvious possibilities are: 1. rebalancing, which happens quite slowly to avoid causing disruption so may take many days to complete 2. relocating shards away from nodes whose disks are nearly full ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 19, 2020, 5:51pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/3 "2020-05-19T17:51:40Z")
disk are not full at all. only 5% used no node has crashed from days. no new index is being introduce. all node has same number of shards. No ILM policy is active ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 19, 2020, 5:53pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/4 "2020-05-19T17:53:48Z")
did more checking found this error [filerequests-2019][4] failed to turn off translog retention org.apache.lucene.store.AlreadyClosedException: engine is closed and same goes for few different index. what is this translog retention? ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 20, 2020, 1:11pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/5 "2020-05-20T13:11:48Z")
I think it is because of this failed to turn off translog retention. how do I fix it? I just show bunch of index becomes yellow on a node, node went offline for while and back to normal after few min. and most all these index are not even being accessed at all for example "filerequests-2016" which I know no one is using. but that one pops up on all log including few others. this is translog stats for that index ``` "translog" : { "operations" : 0, "size_in_bytes" : 2475, "uncommitted_operations" : 0, "uncommitted_size_in_bytes" : 2475, "earliest_last_modified_age" : 0 }, ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 20, 2020, 1:28pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/6 "2020-05-20T13:28:06Z")
No, I do not think that "failed to turn off translog retention" is a cause of relocations, I think this is more likely another symptom of something going wrong. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 20, 2020, 1:39pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/7 "2020-05-20T13:39:14Z")
Should I completely restart whole cluster and check afterword? I have six data node only, five master only and two logstash (which are not running elasticsearch) 142 primary and 142 replica shard. each data node has evenly distributed shard count. each data node has same size of disk, same os, same disk type, same memory ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 20, 2020, 3:16pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/8 "2020-05-20T15:16:47Z")
Can you share some more comprehensive logs? It's not really possible to suggest a course of action from what you've shared so far, there's not enough detail. Any indication of shard failures? ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 20, 2020, 3:28pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/9 "2020-05-20T15:28:50Z")
Hi David what kind of log do you want to see? I can send them no shard failures. cluster goes yellow for few minutes and then becomes green again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 20, 2020, 3:41pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/10 "2020-05-20T15:41:17Z")
I'd like to see the full logs from all nodes from when the cluster was green, through a period of yellow health and back to green again. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 20, 2020, 3:44pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/11 "2020-05-20T15:44:09Z")
OK. let me put myself to work and get that log out and post it ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [May 21, 2020, 1:04pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/12 "2020-05-21T13:04:09Z")
David, I found the problem and fixed yesterday evening. since then cluster has not turn to yellow status while collecting all the logs i show that one of the indice was recreating. Actually I had a cron running which deletes that indice every few hour and at next ingestion logstash creates it. I change that and not dropping indice but just deleting all the document from it. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 21, 2020, 1:56pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/13 "2020-05-21T13:56:43Z")
Yep that'd do it. Nice work, thanks for following up here. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 18, 2020, 2:07pm UTC](https://discuss.elastic.co/t/daily-shard-reallocation/233360/14 "2020-06-18T14:07:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.