# The rebalancing task lasted for more than a week without stopping, and the balance has not been reached

**URL:** <https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020>\
**Category:** Elasticsearch\
**Created:** [September 13, 2021, 8:17am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020 "2021-09-13T08:17:48Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 13, 2021, 8:17am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/1 "2021-09-13T08:17:48Z")

</div>

version ：7.10.2  
My cluster has a total of 30 machines, one machine has two data nodes, so there are a total of 60 nodes  
i add a dataNode to cluster，and then the rebalance tasks started...  
but a week has passed，rebalance keeps going now and not only for new dataNode。  
Will configuring two nodes on one machine cause some bugs in the cluster?Or other issues。  
And i found this situation exists for GET \_cat/recovery?v API :

source\_node target\_node  
nodeA nodeb  
nodeA nodec  
noded nodeA  
nodee nodeA

i tried reduce the cluster\_concurrent\_rebalance and configure rebalance.enable as none for a while，but when i open rebalance ，rebalance tasks continues.....  
Hope someone can help me, thank you very much！

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 3:25am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/2 "2021-09-14T03:25:41Z")

</div>

Is there any official person to help me? 😭 I think this is like a bug, but I haven’t found a specific version that will fix it.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 14, 2021, 3:38am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/3 "2021-09-14T03:38:38Z")

</div>

What's the output from `_cat/allocattion?v`?  
Do you have any allocation awareness settings applied?  
What do your master node logs show?

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 4:12am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/4 "2021-09-14T04:12:09Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/0/8/082846c03e88b510c5c13ce662124ddafb70a2d3.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18caa41872a2463cbb7a67f0989e6d7923ebca60.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91d1c2442573a59a7ca4b11486c2d35374efa200.jpeg)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 14, 2021, 4:21am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/5 "2021-09-14T04:21:57Z")

</div>

Please don't post pictures of text or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 🙂

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 4:45am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/6 "2021-09-14T04:45:07Z")

</div>

The following is the master log：

```auto
[2021-09-14T01:51:00,001][INFO][o.e.x.m.MlDailyMaintenanceService] [master-xxx.xxx.xxx.xxx-9210] triggering scheduled [ML] maintenance tasks
[2021-09-14T01:51:00,023][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [master-xxx.xxx.xxx.xxx-9210] Deleting expired data
[2021-09-14T01:51:00,127][INFO][o.e.x.m.j.r.UnusedStatsRemover] [master-xxx.xxx.xxx.xxx-9210] Successfully deleted [0] unused stats documents
[2021-09-14T01:51:00,128][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [master-xxx.xxx.xxx.xxx-9210] Completed deletion of expired ML data
[2021-09-14T01:51:00,128][INFO][o.e.x.m.MlDailyMaintenanceService] [master-xxx.xxx.xxx.xxx-9210] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2021-09-14T06:05:01,138][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.12-000040] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [metricbeat]
[2021-09-14T06:05:01,443][INFO][o.e.c.m.MetadataCreateIndexService] [master-xxx.xxx.xxx.xxx-9210] [metricbeat-7.10.2-2021.09.13-000041] creating index, cause [rollover_index], templates [metricbeat-7.10.2], shards [1]/[1]
[2021-09-14T06:05:03,538][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.13-000041] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [metricbeat]
[2021-09-14T06:05:03,757][INFO][o.e.c.m.MetadataMappingService] [master-xxx.xxx.xxx.xxx-9210] [metricbeat-7.10.2-2021.09.13-000041/ph9UaWOuSv6dr1LqaLbQlA] update_mapping [_doc]
[2021-09-14T06:05:05,053][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.12-000040] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [metricbeat]
[2021-09-14T06:05:05,271][INFO][o.e.c.m.MetadataMappingService] [master-xxx.xxx.xxx.xxx-9210] [metricbeat-7.10.2-2021.09.13-000041/ph9UaWOuSv6dr1LqaLbQlA] update_mapping [_doc]
[2021-09-14T06:05:06,831][INFO][o.e.c.m.MetadataMappingService] [master-xxx.xxx.xxx.xxx-9210] [metricbeat-7.10.2-2021.09.13-000041/ph9UaWOuSv6dr1LqaLbQlA] update_mapping [_doc]
[2021-09-14T06:05:07,195][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.13-000041] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [metricbeat]
[2021-09-14T06:05:07,326][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.12-000040] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [metricbeat]
[2021-09-14T06:05:07,344][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.12-000040] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [metricbeat]
[2021-09-14T06:05:08,339][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.13-000041] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [metricbeat]
[2021-09-14T06:05:08,456][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [metricbeat-7.10.2-2021.09.12-000040] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [metricbeat]
[2021-09-14T06:09:21,953][INFO][o.e.c.r.a.AllocationService] [master-xxx.xxx.xxx.xxx-9210] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[metricbeat-7.10.2-2021.09.13-000041][0]]]).

```

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 4:48am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/7 "2021-09-14T04:48:57Z")

</div>

Regarding the configuration of allocation, I only modified the following three：

```auto
cluster.routing.allocation.balance.index ：0.45 （default：0.55）
cluster.routing.allocation.balance.shard：0.55 （default：0.45）
cluster.routing.allocation.cluster_concurrent_rebalance：40 （default：2）

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 14, 2021, 4:51am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/8 "2021-09-14T04:51:38Z")

</div>

There's nothing in your logs that shows reallocation other than what would be expected from an ILM policy, so it's hard to see what's happening.

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 4:54am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/9 "2021-09-14T04:54:19Z")

</div>

The output from `_cat/allocattion?v`are as follows：（I have hidden the content of ip and host）

```auto
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
   273 3.2tb 3.3tb 2.5tb 5.8tb 56 
   273 3.1tb 3.1tb 2.6tb 5.8tb 54 
   262 2.9tb 2.9tb 2.8tb 5.8tb 50 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3tb 3.1tb 2.6tb 5.8tb 53
   274 3.2tb 3.2tb 2.5tb 5.8tb 55 
   274 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.1tb 3.1tb 2.7tb 5.8tb 53 
   214 1.7tb 1.7tb 4tb 5.8tb 29 
   274 3tb 3tb 2.7tb 5.8tb 52 
   273 3.2tb 3.2tb 2.5tb 5.8tb 56
   273 3tb 3tb 2.7tb 5.8tb 52 
   272 3.1tb 3.1tb 2.6tb 5.8tb 54 
   271 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.3tb 3.3tb 2.4tb 5.8tb 58 
   274 3.4tb 3.4tb 2.3tb 5.8tb 58
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.4tb 3.4tb 2.3tb 5.8tb 59
   272 3.2tb 3.2tb 2.5tb 5.8tb 56
   272 3.2tb 3.2tb 2.5tb 5.8tb 56 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   274 3.2tb 3.2tb 2.5tb 5.8tb 56 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   272 3.1tb 3.2tb 2.6tb 5.8tb 55 
   273 3.2tb 3.2tb 2.5tb 5.8tb 56 
   273 3.2tb 3.2tb 2.5tb 5.8tb 55
   274 3.1tb 3.1tb 2.6tb 5.8tb 54 
   273 3.1tb 3.1tb 2.6tb 5.8tb 54
   264 3.4tb 3.4tb 2.3tb 5.8tb 59
   272 3.4tb 3.4tb 2.4tb 5.8tb 58 
   270 3tb 3tb 2.7tb 5.8tb 52 
   273 3.1tb 3.1tb 2.6tb 5.8tb 54
   273 3.2tb 3.3tb 2.4tb 5.8tb 57
   273 3.2tb 3.2tb 2.5tb 5.8tb 55
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.1tb 3.1tb 2.6tb 5.8tb 54
   273 3.2tb 3.2tb 2.5tb 5.8tb 55 
   273 3.2tb 3.2tb 2.5tb 5.8tb 56
   272 3.3tb 3.3tb 2.4tb 5.8tb 57 
   253 2.6tb 2.7tb 3tb 5.8tb 46
   273 3.1tb 3.1tb 2.6tb 5.8tb 54 
   213 2.2tb 2.2tb 3.6tb 5.8tb 38
   275 3.1tb 3.1tb 2.7tb 5.8tb 53 
   274 3.2tb 3.2tb 2.5tb 5.8tb 56 
   273 3.4tb 3.4tb 2.3tb 5.8tb 59 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57 
   273 3.3tb 3.3tb 2.4tb 5.8tb 57
   275 3.3tb 3.3tb 2.4tb 5.8tb 57
   273 3.3tb 3.3tb 2.5tb 5.8tb 56 
   271 2.8tb 2.8tb 2.9tb 5.8tb 49 
   273 3.3tb 3.3tb 2.4tb 5.8tb 58
   273 3.2tb 3.2tb 2.5tb 5.8tb 56
   257 3.2tb 3.2tb 2.5tb 5.8tb 55
   273 3.2tb 3.2tb 2.5tb 5.8tb 55 
   265 3.3tb 3.3tb 2.4tb 5.8tb 58
   274 3.3tb 3.3tb 2.4tb 5.8tb 57
   274 3tb 3tb 2.7tb 5.8tb 53 
   273 3.2tb 3.2tb 2.5tb 5.8tb 56 
   265 2.9tb 2.9tb 2.8tb 5.8tb 50 
   273 3.2tb 3.2tb 2.5tb 5.8tb 55 
   273 3.3tb 3.3tb 2.5tb 5.8tb 56

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2021, 4:57am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/10 "2021-09-14T04:57:56Z")

</div>

What is the full output of the cluster stats API?

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 4:59am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/11 "2021-09-14T04:59:30Z")

</div>

This is indeed all the logs, and the newly generated logs are basically similar to the following：  
`[2021-09-14T11:25:05,106][INFO][o.e.x.i.IndexLifecycleTransition] [master-xxx.xxx.xxx.xxx-9210] moving index [.kibana-event-log-7.10.2-000007] from [{"phase":"hot","action":"unfollow","name":"wait-for-yellow-step"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [kibana-event-log-policy]`

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2021, 5:03am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/12 "2021-09-14T05:03:11Z")

</div>

That message indicates that the index is moved to a different state, not a different node. You can monitor shard movements using the [cat recovery API](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-recovery.html). What does this show?

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 5:05am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/13 "2021-09-14T05:05:28Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> cluster stats

Here is the full output of the cluster stats：

```auto
{
    "_nodes": {
        "total": 70,
        "successful": 70,
        "failed": 0
    },
    "cluster_name": "xxxxx",
    "cluster_uuid": "xxxxx",
    "timestamp": 1631595738148,
    "status": "green",
    "indices": {
        "count": 1391,
        "shards": {
            "total": 16735,
            "primaries": 8379,
            "replication": 0.9972550423678244,
            "index": {
                "shards": {
                    "min": 1,
                    "max": 400,
                    "avg": 12.030913012221424
                },
                "primaries": {
                    "min": 1,
                    "max": 200,
                    "avg": 6.02372393961179
                },
                "replication": {
                    "min": 0,
                    "max": 1,
                    "avg": 0.9971243709561467
                }
            }
        },
        "docs": {
            "count": 73134894750,
            "deleted": 592805409
        },
        "store": {
            "size_in_bytes": 217492731193541,
            "reserved_in_bytes": 0
        },
        "fielddata": {
            "memory_size_in_bytes": 4419829176,
            "evictions": 0
        },
        "query_cache": {
            "memory_size_in_bytes": 15333618230,
            "total_count": 66970368294,
            "hit_count": 427855687,
            "miss_count": 66542512607,
            "cache_size": 153131,
            "cache_count": 1190658,
            "evictions": 1037527
        },
        "completion": {
            "size_in_bytes": 0
        },
        "segments": {
            "count": 83237,
            "memory_in_bytes": 2224512332,
            "terms_memory_in_bytes": 1287050168,
            "stored_fields_memory_in_bytes": 71620168,
            "term_vectors_memory_in_bytes": 488,
            "norms_memory_in_bytes": 35990080,
            "points_memory_in_bytes": 0,
            "doc_values_memory_in_bytes": 829851428,
            "index_writer_memory_in_bytes": 5768524576,
            "version_map_memory_in_bytes": 10630400,
            "fixed_bit_set_memory_in_bytes": 7579813112,
            "max_unsafe_auto_id_timestamp": 1631589312224,
            "file_sizes": {}
        },
        "mappings": {
            "field_types": [
                {
                    "name": "alias",
                    "count": 123,
                    "index_count": 41
                },
                {
                    "name": "binary",
                    "count": 15,
                    "index_count": 4
                },
                {
                    "name": "boolean",
                    "count": 9888,
                    "index_count": 1272
                },
                {
                    "name": "byte",
                    "count": 42,
                    "index_count": 42
                },
                {
                    "name": "date",
                    "count": 8904,
                    "index_count": 1382
                },
                {
                    "name": "date_nanos",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "date_range",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "dense_vector",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "double",
                    "count": 7678,
                    "index_count": 1204
                },
                {
                    "name": "double_range",
                    "count": 3,
                    "index_count": 2
                },
                {
                    "name": "flattened",
                    "count": 9,
                    "index_count": 1
                },
                {
                    "name": "float",
                    "count": 8519,
                    "index_count": 117
                },
                {
                    "name": "float_range",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "geo_point",
                    "count": 292,
                    "index_count": 45
                },
                {
                    "name": "geo_shape",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "half_float",
                    "count": 69,
                    "index_count": 16
                },
                {
                    "name": "integer",
                    "count": 191,
                    "index_count": 14
                },
                {
                    "name": "integer_range",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "ip",
                    "count": 821,
                    "index_count": 42
                },
                {
                    "name": "ip_range",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "keyword",
                    "count": 122970,
                    "index_count": 1387
                },
                {
                    "name": "long",
                    "count": 121073,
                    "index_count": 1373
                },
                {
                    "name": "long_range",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "nested",
                    "count": 998,
                    "index_count": 697
                },
                {
                    "name": "object",
                    "count": 99352,
                    "index_count": 450
                },
                {
                    "name": "scaled_float",
                    "count": 4961,
                    "index_count": 41
                },
                {
                    "name": "shape",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "short",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "text",
                    "count": 14365,
                    "index_count": 1371
                }
            ]
        },
        "analysis": {
            "char_filter_types": [],
            "tokenizer_types": [],
            "filter_types": [
                {
                    "name": "pattern_capture",
                    "count": 1,
                    "index_count": 1
                }
            ],
            "analyzer_types": [
                {
                    "name": "custom",
                    "count": 1,
                    "index_count": 1
                }
            ],
            "built_in_char_filters": [],
            "built_in_tokenizers": [
                {
                    "name": "uax_url_email",
                    "count": 1,
                    "index_count": 1
                }
            ],
            "built_in_filters": [
                {
                    "name": "lowercase",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "unique",
                    "count": 1,
                    "index_count": 1
                }
            ],
            "built_in_analyzers": [
                {
                    "name": "english",
                    "count": 1,
                    "index_count": 1
                },
                {
                    "name": "ik_max_word",
                    "count": 10162,
                    "index_count": 1238
                }
            ]
        }
    },
    "nodes": {
        "count": {
            "total": 70,
            "coordinating_only": 4,
            "data": 0,
            "data_cold": 0,
            "data_content": 62,
            "data_hot": 0,
            "data_warm": 62,
            "ingest": 1,
            "master": 3,
            "ml": 0,
            "remote_cluster_client": 0,
            "transform": 0,
            "voting_only": 0
        },
        "versions": [
            "7.10.2"
        ],
        "os": {
            "available_processors": 5216,
            "allocated_processors": 5216,
            "names": [
                {
                    "name": "Linux",
                    "count": 70
                }
            ],
            "pretty_names": [
                {
                    "pretty_name": "CentOS Linux 7 (Core)",
                    "count": 70
                }
            ],
            "mem": {
                "total_in_bytes": 17807501201408,
                "free_in_bytes": 897589530624,
                "used_in_bytes": 16909911670784,
                "free_percent": 5,
                "used_percent": 95
            }
        },
        "process": {
            "cpu": {
                "percent": 221
            },
            "open_file_descriptors": {
                "min": 1887,
                "max": 3758,
                "avg": 3469
            }
        },
        "jvm": {
            "max_uptime_in_millis": 1611117676,
            "versions": [
                {
                    "version": "15.0.1",
                    "vm_name": "OpenJDK 64-Bit Server VM",
                    "vm_version": "15.0.1+9",
                    "vm_vendor": "AdoptOpenJDK",
                    "bundled_jdk": true,
                    "using_bundled_jdk": true,
                    "count": 70
                }
            ],
            "mem": {
                "heap_used_in_bytes": 1028837157904,
                "heap_max_in_bytes": 2195751698432
            },
            "threads": 24368
        },
        "fs": {
            "total_in_bytes": 201363506065408,
            "free_in_bytes": 91212150325248,
            "available_in_bytes": 91212150325248
        },
        "plugins": [
            {
                "name": "analysis-ik",
                "version": "7.10.2",
                "elasticsearch_version": "7.10.2",
                "java_version": "1.8",
                "description": "IK Analyzer for Elasticsearch",
                "classname": "org.elasticsearch.plugin.analysis.ik.AnalysisIkPlugin",
                "extended_plugins": [],
                "has_native_controller": false
            }
        ],
        "network_types": {
            "transport_types": {
                "security4": 70
            },
            "http_types": {
                "security4": 70
            }
        },
        "discovery_types": {
            "zen": 70
        },
        "packaging_types": [
            {
                "flavor": "default",
                "type": "tar",
                "count": 70
            }
        ],
        "ingest": {
            "number_of_pipelines": 18,
            "processor_stats": {
                "conditional": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "geoip": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "grok": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "gsub": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "remove": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "rename": {
                    "count": 0,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 0
                },
                "script": {
                    "count": 14,
                    "failed": 4,
                    "current": 0,
                    "time_in_millis": 21
                },
                "set": {
                    "count": 10,
                    "failed": 0,
                    "current": 0,
                    "time_in_millis": 18
                }
            }
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 5:13am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/14 "2021-09-14T05:13:25Z")

</div>

This is also troubled by me, so I can't find the reason 😭

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 6:36am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/15 "2021-09-14T06:36:22Z")

</div>

Here: 😄

```auto
sl_weixin-comment_202011_v1 0 1.3m peer index ip21 data-ip21-9201 ip15 data-ip15-9201 n/a n/a 27 26 96.3% 27 4584744775 3243567409 70.7% 4584744775 1 0 0.0%
sl_weixin_201812_v1 7 4.8m peer index ip27 data-ip27-9202 ip21 data-ip21-9202 n/a n/a 18 16 88.9% 18 28098184802 12458809480 44.3% 28098184802 0 0 100.0%
sl_video_202105_v1 0 2.5m peer index ip24 data-ip24-9201 ip13 data-ip13-9201 n/a n/a 45 43 95.6% 45 22960400828 12584962168 54.8% 22960400828 0 0 100.0%
sl_video_202105_v1 3 3m peer index ip4 data-ip4-9201 ip31 data-ip31-9202 n/a n/a 51 48 94.1% 51 22967151236 5357819127 23.3% 22967151236 0 0 100.0%
sl_video_202105_v1 6 13.1m peer index ip8 data-ip8-9202 ip31 data-ip31-9202 n/a n/a 42 41 97.6% 42 22964004446 21265061227 92.6% 22964004446 0 0 100.0%
sl_video_202105_v1 8 18.4s peer index ip27 data-ip27-9202 ip16 data-ip16-9201 n/a n/a 54 49 90.7% 54 22963282082 685035437 3.0% 22963282082 0 0 100.0%
sl_video_202105_v1 9 4.4m peer index ip22 data-ip22-9202 ip30 data-ip30-9201 n/a n/a 51 50 98.0% 51 22959658140 18463867294 80.4% 22959658140 0 0 100.0%
sl_video_202105_v1 10 4.9m peer index ip12 data-ip12-9202 ip5 data-ip5-9202 n/a n/a 36 35 97.2% 36 22957972007 15950452035 69.5% 22957972007 0 0 100.0%
sl_video_202105_v1 11 4.4m peer index ip2 data-ip2-9201 ip13 data-ip13-9202 n/a n/a 30 29 96.7% 30 22964789531 15874673342 69.1% 22964789531 0 0 100.0%
sl_video_202105_v1 13 1.1m peer index ip3 data-ip3-9202 ip15 data-ip15-9201 n/a n/a 36 32 88.9% 36 22970007002 2324170178 10.1% 22970007002 0 0 100.0%
sl_video_202105_v1 19 2.6m peer index ip3 data-ip3-9201 ip9 data-ip9-9201 n/a n/a 51 49 96.1% 51 22960550766 8407490986 36.6% 22960550766 0 0 100.0%
sl_news-app_202002_v1 5 9.9m peer index ip15 data-ip15-9201 ip23 data-ip23-9201 n/a n/a 18 17 94.4% 18 25792120984 21180013805 82.1% 25792120984 0 0 100.0%
sl_news_202108_v1 8 2.9m peer index ip11 data-ip11-9201 ip3 data-ip3-9201 n/a n/a 30 28 93.3% 30 25186986014 10513747439 41.7% 25186986014 397 0 0.0%
sl_news_202108_v1 16 2.1m peer index ip4 data-ip4-9201 ip31 data-ip31-9201 n/a n/a 48 45 93.8% 48 25160602090 3617682820 14.4% 25160602090 902 0 0.0%
user_douyin_v1 1 56.6s peer index ip23 data-ip23-9201 ip25 data-ip25-9201 n/a n/a 209 200 95.7% 209 25587485452 2727847757 10.7% 25587485452 336570 0 0.0%
sl_weibo_201906_v1 8 34.3s peer index ip11 data-ip11-9202 ip9 data-ip9-9201 n/a n/a 51 47 92.2% 51 23465130975 2414661724 10.3% 23465130975 255 0 0.0%
sl_weibo_201906_v1 16 6.8m peer index ip14 data-ip14-9202 ip16 data-ip16-9202 n/a n/a 39 38 97.4% 39 23478757865 19985333192 85.1% 23478757865 1294 0 0.0%
sl_video_201906_v1 0 1.9m peer index ip6 data-ip6-9201 ip5 data-ip5-9202 n/a n/a 18 15 83.3% 18 15343534060 4666299323 30.4% 15343534060 0 0 100.0%
sl_weibo_202108_v1 17 5.5m peer index ip16 data-ip16-9202 ip22 data-ip22-9202 n/a n/a 183 182 99.5% 183 22703078331 15183121998 66.9% 22703078331 1397 0 0.0%
sl_weibo_202108_v1 21 7.5m peer index ip16 data-ip16-9201 ip23 data-ip23-9201 n/a n/a 174 173 99.4% 174 23268438750 15040598115 64.6% 23268438750 1829 0 0.0%
sl_weibo_202108_v1 23 5.7m peer index ip14 data-ip14-9202 ip5 data-ip5-9201 n/a n/a 177 176 99.4% 177 22618664665 17908983247 79.2% 22618664665 38323 0 0.0%
sl_weibo_202108_v1 41 6.6m peer index ip2 data-ip2-9201 ip18 data-ip18-9201 n/a n/a 126 125 99.2% 126 22725646516 17332080221 76.3% 22725646516 2040 0 0.0%
sl_weibo_202108_v1 45 4m peer index ip3 data-ip3-9201 ip31 data-ip31-9201 n/a n/a 180 176 97.8% 180 22652925301 4435401149 19.6% 22652925301 15416 0 0.0%
sl_ec_201908_v1 3 9.2m peer index ip16 data-ip16-9202 ip3 data-ip3-9202 n/a n/a 39 38 97.4% 39 20680485901 20458800078 98.9% 20680485901 0 0 100.0%
sl_ec_201908_v1 4 1.5m peer index ip13 data-ip13-9201 ip7 data-ip7-9202 n/a n/a 33 30 90.9% 33 20679950361 5864422857 28.4% 20679950361 0 0 100.0%
sl_weixin_201908_v1 9 7.4m peer index ip28 data-ip28-9202 ip21 data-ip21-9202 n/a n/a 18 17 94.4% 18 25511425056 24435084882 95.8% 25511425056 0 0 100.0%
sl_ec_202104_v1 1 2.8m peer index ip5 data-ip5-9202 ip8 data-ip8-9201 n/a n/a 60 57 95.0% 60 23983263781 7085794663 29.5% 23983263781 8 0 0.0%
sl_video_201912_v1 0 2.8m peer index ip8 data-ip8-9201 ip28 data-ip28-9201 n/a n/a 39 36 92.3% 39 34673379128 10229684234 29.5% 34673379128 0 0 100.0%
sl_news_202007_v1 8 3.9m peer index ip23 data-ip23-9202 ip30 data-ip30-9202 n/a n/a 18 16 88.9% 18 26953828769 11607792126 43.1% 26953828769 0 0 100.0%
sl_weibo_202102_v1 17 1.4m peer index ip30 data-ip30-9201 ip23 data-ip23-9202 n/a n/a 69 65 94.2% 69 25007976295 4883245308 19.5% 25007976295 5122 0 0.0%
sl_news_201910_v1 2 3.3m peer index ip1 data-ip1-9201 ip26 data-ip26-9202 n/a n/a 18 17 94.4% 18 21022294710 13656217903 65.0% 21022294710 0 0 100.0%
sl_weibo-comment_201812_v1 0 1.8m peer index ip20 data-ip20-9201 ip14 data-ip14-9202 n/a n/a 18 15 83.3% 18 23291417512 4988002368 21.4% 23291417512 0 0 100.0%
kol_user_v1 0 24.4s peer index ip5 data-ip5-9202 ip17 data-ip17-9202 n/a n/a 125 115 92.0% 125 3546976610 742052567 20.9% 3546976610 0 0 100.0%
sl_news-app_201908_v1 0 7.3m peer index ip30 data-ip30-9202 ip25 data-ip25-9201 n/a n/a 18 17 94.4% 18 25462553741 18340864940 72.0% 25462553741 0 0 100.0%
sl_news_202012_v1 19 4.6m peer index ip29 data-ip29-9202 ip18 data-ip18-9201 n/a n/a 18 16 88.9% 18 24987389909 13312075006 53.3% 24987389909 0 0 100.0%
sl_weibo_202107_v1 48 1.8m peer index ip15 data-ip15-9202 ip2 data-ip2-9202 n/a n/a 81 79 97.5% 81 27962643024 11847502839 42.4% 27962643024 4804 0 0.0%
sl_ec_201909_v1 5 3.8m peer index ip18 data-ip18-9201 ip26 data-ip26-9202 n/a n/a 66 64 97.0% 66 21500142782 10109975061 47.0% 21500142782 0 0 100.0%
sl_news-app_202004_v1 13 11.1m peer index ip16 data-ip16-9201 ip28 data-ip28-9201 n/a n/a 18 17 94.4% 18 25903099528 22495888780 86.8% 25903099528 0 0 100.0%

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 14, 2021, 10:32am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/16 "2021-09-14T10:32:46Z")

</div>

> [@liwanjie](#):
>
> ```auto
> cluster.routing.allocation.cluster_concurrent_rebalance：40 （default：2）
> 
> ```

This is almost certainly a bad idea. This setting controls how far ahead the balancer looks, but the lookahead isn't perfect so if you set it very high (i.e. `40`) it's probably overshooting and then having to back-track. Set it back to `2`.

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 14, 2021, 11:05am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/17 "2021-09-14T11:05:35Z")

</div>

Thank you very much.  
I will set it to `2` to see if it works

---

<div class="post-metadata">

**Author:** ![liwanjie](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@liwanjie](https://discuss.elastic.co/u/liwanjie)\
**Post date:** [September 15, 2021, 2:32am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/18 "2021-09-15T02:32:51Z")

</div>

I restarted the current master node of the cluster yesterday  
Now that the cluster is balanced  
Feels very inexplicable  
Thank you very much for answering my questions and helping me

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 13, 2021, 2:33am UTC](https://discuss.elastic.co/t/the-rebalancing-task-lasted-for-more-than-a-week-without-stopping-and-the-balance-has-not-been-reached/284020/19 "2021-10-13T02:33:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.