# Auto\_expand\_replicas not working on enrich index

**URL:** <https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569>\
**Category:** Elasticsearch\
**Created:** [September 2, 2022, 4:35pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569 "2022-09-02T16:35:11Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 2, 2022, 4:35pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/1 "2022-09-02T16:35:11Z")

</div>

Hi guys,

Our enrich indices are not replicated on our production cluster, and I don't understand why... Can you help to troubleshoot the issue?

Context

- ES 7.17.1
- 3 Nodes
- Many various data indices
- Many ingest pipelines using enrich processors
- Many enrich policies

Nodes

```auto
GET /_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
- 50 100 10 4.01 3.55 3.61 cdfhilmrstw * es-03.stordata.dc
- 57 98 7 4.11 3.91 4.01 cdfhilmrstw - es-01.stordata.dc
- 49 94 7 5.34 4.42 4.09 cdfhilmrstw - es-02.stordata.dc

```

Cluster settings

```auto
GET _cluster/settings
{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "cluster_concurrent_rebalance" : "16",
          "node_concurrent_recoveries" : "4",
          "disk" : {
            "watermark" : {
              "low" : "240gb",
              "flood_stage" : "80gb",
              "high" : "120gb"
            }
          },
          "node_initial_primaries_recoveries" : "8"
        }
      },
      "max_shards_per_node" : "3000"
    },
    "indices" : {
      "lifecycle" : {
        "poll_interval" : "1m"
      },
      "recovery" : {
        "max_bytes_per_sec" : "480mb"
      }
    },
    "search" : {
      "max_buckets" : "100000"
    },
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "false"
          }
        },
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

```

Example index where auto\_expand\_replicas is not working

```auto
GET /.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d/_settings
#! this request accesses system indices: [.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167], but in a future major version, direct access to system indices will be prevented by default
{
  ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "refresh_interval" : "-1",
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167",
        "max_inner_result_window" : "10000",
        "creation_date" : "1662134874167",
        "number_of_replicas" : "0",
        "uuid" : "mjvooklZSxS2b5YWKxX9Pw",
        "version" : {
          "created" : "7170199"
        },
        "warmer" : {
          "enabled" : "false"
        }
      }
    }
  }
}

```

Trying to manually update number\_of\_replicas immediately turn it back to `0`.  
The problem is also there on other system indices like `.kibana`, `.monitoring`, `.slm-history`, etc. and seems like a general problem on the cluster...

Side note: is there some logging I could enable to further track down the decision to not replicate the shards?

Thanks in advance,  
David

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 2, 2022, 5:22pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/2 "2022-09-02T17:22:37Z")

</div>

> [@ddolcimascolo](#):
>
> `"_tier_preference" : "data_content"`

Are all nodes set to `data_content`?

> [@ddolcimascolo](#):
>
> `"number_of_replicas" : "0",`

I believe auto\_expand\_replicas should override this, but have you checked if changing it makes any difference?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 2, 2022, 5:48pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/3 "2022-09-02T17:48:03Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Are all nodes set to `data_content`?

We don't use data tiers at all, so nodes are setup with `node.roles: []` and I believe that they get all roles in this case.

> [@Christian\_Dahlqvist](#):
>
> I believe auto\_expand\_replicas should override this, but have you checked if changing it makes any difference?

Yep, I mentioned this in the original post. Setting it immediately revert it back to `0`, probably in line with the decision to assign 0 replicas...

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 5, 2022, 9:16am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/4 "2022-09-05T09:16:49Z")

</div>

Hi everyone,

Is anybody able to help ?

Cheers,  
David

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 5, 2022, 1:10pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/5 "2022-09-05T13:10:49Z")

</div>

> [@ddolcimascolo](#):
>
> We don't use data tiers at all, so nodes are setup with `node.roles: []` and I believe that they get all roles in this case.

`node.roles: []` means the node has no roles. That'd explain it I think, this index requires a `data_content` node so if you have no such nodes then it will auto-expand to zero replicas.

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 5, 2022, 1:38pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/6 "2022-09-05T13:38:08Z")

</div>

Yep sorry. I mean nodes don't have the `node.roles` attribute at all, thus having all roles. You can confirm that by the `_cat/nodes` I copied in my original post

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 5, 2022, 1:42pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/7 "2022-09-05T13:42:52Z")

</div>

Hmm ok could you run the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) on this shard and share the full response here?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 5, 2022, 2:46pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/8 "2022-09-05T14:46:18Z")

</div>

So the index has a single primary shard, I ran the API on it

```auto
GET _cluster/allocation/explain
{
  "index": ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662386868413",
  "shard": 0,
  "primary": true
}

{
  "index" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662386868413",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "PS1WfPGWS4KvFDJbACEG4A",
    "name" : "es-01.stordata.dc",
    "transport_address" : "10.254.249.10:9300",
    "attributes" : {
      "ml.machine_memory" : "135075225600",
      "ml.max_open_jobs" : "512",
      "xpack.installed" : "true",
      "ml.max_jvm_size" : "33285996544",
      "transform.node" : "true"
    },
    "weight_ranking" : 1
  },
  "can_remain_on_current_node" : "yes",
  "can_rebalance_cluster" : "yes",
  "can_rebalance_to_other_node" : "no",
  "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
  "node_allocation_decisions" : [
    {
      "node_id" : "NeLqOXVAQa-38GN1jWyn_Q",
      "node_name" : "es-03.stordata.dc",
      "transport_address" : "10.254.249.12:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075368960",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1
    },
    {
      "node_id" : "qZMbs-d_QcOGh3Ve2DS1ug",
      "node_name" : "es-02.stordata.dc",
      "transport_address" : "10.254.249.11:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075360768",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 5, 2022, 3:06pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/9 "2022-09-05T15:06:22Z")

</div>

Weird, that says it should be able to expand to all 3 nodes. Could you try again with `GET _cluster/allocation/explain?include_yes_decisions` to get a bit more detail?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 5, 2022, 3:33pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/10 "2022-09-05T15:33:02Z")

</div>

Sure, thx for helping out btw !

```auto
{
  "index" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662390468940",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "PS1WfPGWS4KvFDJbACEG4A",
    "name" : "es-01.stordata.dc",
    "transport_address" : "10.254.249.10:9300",
    "attributes" : {
      "ml.machine_memory" : "135075225600",
      "ml.max_open_jobs" : "512",
      "xpack.installed" : "true",
      "ml.max_jvm_size" : "33285996544",
      "transform.node" : "true"
    },
    "weight_ranking" : 1
  },
  "can_remain_on_current_node" : "yes",
  "can_rebalance_cluster" : "yes",
  "can_rebalance_to_other_node" : "no",
  "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
  "node_allocation_decisions" : [
    {
      "node_id" : "NeLqOXVAQa-38GN1jWyn_Q",
      "node_name" : "es-03.stordata.dc",
      "transport_address" : "10.254.249.12:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075368960",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "YES",
          "explanation" : "shard has no previous failures"
        },
        {
          "decider" : "replica_after_primary_active",
          "decision" : "YES",
          "explanation" : "shard is primary and can be allocated"
        },
        {
          "decider" : "enable",
          "decision" : "YES",
          "explanation" : "all allocations are allowed"
        },
        {
          "decider" : "node_version",
          "decision" : "YES",
          "explanation" : "can relocate primary shard from a node with version [7.17.1] to a node with equal-or-newer version [7.17.1]"
        },
        {
          "decider" : "snapshot_in_progress",
          "decision" : "YES",
          "explanation" : "no snapshots are currently running"
        },
        {
          "decider" : "restore_in_progress",
          "decision" : "YES",
          "explanation" : "ignored as shard is not being recovered from a snapshot"
        },
        {
          "decider" : "node_shutdown",
          "decision" : "YES",
          "explanation" : "this node is not currently shutting down"
        },
        {
          "decider" : "node_replacement",
          "decision" : "YES",
          "explanation" : "neither the source nor target node are part of an ongoing node replacement (no replacements)"
        },
        {
          "decider" : "filter",
          "decision" : "YES",
          "explanation" : "node passes include/exclude/require filters"
        },
        {
          "decider" : "same_shard",
          "decision" : "YES",
          "explanation" : "this node does not hold a copy of this shard"
        },
        {
          "decider" : "disk_threshold",
          "decision" : "YES",
          "explanation" : "enough disk for shard on node, free: [1.4tb], shard size: [207kb], free after allocating shard: [1.4tb]"
        },
        {
          "decider" : "throttling",
          "decision" : "YES",
          "explanation" : "below shard recovery limit of outgoing: [0 < 4] incoming: [0 < 4]"
        },
        {
          "decider" : "shards_limit",
          "decision" : "YES",
          "explanation" : "total shard limits are disabled: [index: -1, cluster: -1] <= 0"
        },
        {
          "decider" : "awareness",
          "decision" : "YES",
          "explanation" : "allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it"
        },
        {
          "decider" : "data_tier",
          "decision" : "YES",
          "explanation" : "index has a preference for tiers [data_content] and node has tier [data_content]"
        },
        {
          "decider" : "ccr_primary_follower",
          "decision" : "YES",
          "explanation" : "shard is not a follower and is not under the purview of this decider"
        },
        {
          "decider" : "searchable_snapshots",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshot_repository_exists",
          "decision" : "YES",
          "explanation" : "this decider only applies to indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshots_enable",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "dedicated_frozen_node",
          "decision" : "YES",
          "explanation" : "this node's data roles are not exactly [data_frozen] so it is not a dedicated frozen node"
        }
      ]
    },
    {
      "node_id" : "qZMbs-d_QcOGh3Ve2DS1ug",
      "node_name" : "es-02.stordata.dc",
      "transport_address" : "10.254.249.11:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075360768",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "YES",
          "explanation" : "shard has no previous failures"
        },
        {
          "decider" : "replica_after_primary_active",
          "decision" : "YES",
          "explanation" : "shard is primary and can be allocated"
        },
        {
          "decider" : "enable",
          "decision" : "YES",
          "explanation" : "all allocations are allowed"
        },
        {
          "decider" : "node_version",
          "decision" : "YES",
          "explanation" : "can relocate primary shard from a node with version [7.17.1] to a node with equal-or-newer version [7.17.1]"
        },
        {
          "decider" : "snapshot_in_progress",
          "decision" : "YES",
          "explanation" : "no snapshots are currently running"
        },
        {
          "decider" : "restore_in_progress",
          "decision" : "YES",
          "explanation" : "ignored as shard is not being recovered from a snapshot"
        },
        {
          "decider" : "node_shutdown",
          "decision" : "YES",
          "explanation" : "node [qZMbs-d_QcOGh3Ve2DS1ug] is preparing to restart, but will remain in the cluster"
        },
        {
          "decider" : "node_replacement",
          "decision" : "YES",
          "explanation" : "neither the source nor target node are part of an ongoing node replacement (no replacements)"
        },
        {
          "decider" : "filter",
          "decision" : "YES",
          "explanation" : "node passes include/exclude/require filters"
        },
        {
          "decider" : "same_shard",
          "decision" : "YES",
          "explanation" : "this node does not hold a copy of this shard"
        },
        {
          "decider" : "disk_threshold",
          "decision" : "YES",
          "explanation" : "enough disk for shard on node, free: [1.4tb], shard size: [207kb], free after allocating shard: [1.4tb]"
        },
        {
          "decider" : "throttling",
          "decision" : "YES",
          "explanation" : "below shard recovery limit of outgoing: [0 < 4] incoming: [0 < 4]"
        },
        {
          "decider" : "shards_limit",
          "decision" : "YES",
          "explanation" : "total shard limits are disabled: [index: -1, cluster: -1] <= 0"
        },
        {
          "decider" : "awareness",
          "decision" : "YES",
          "explanation" : "allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it"
        },
        {
          "decider" : "data_tier",
          "decision" : "YES",
          "explanation" : "index has a preference for tiers [data_content] and node has tier [data_content]"
        },
        {
          "decider" : "ccr_primary_follower",
          "decision" : "YES",
          "explanation" : "shard is not a follower and is not under the purview of this decider"
        },
        {
          "decider" : "searchable_snapshots",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshot_repository_exists",
          "decision" : "YES",
          "explanation" : "this decider only applies to indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshots_enable",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "dedicated_frozen_node",
          "decision" : "YES",
          "explanation" : "this node's data roles are not exactly [data_frozen] so it is not a dedicated frozen node"
        }
      ]
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 5, 2022, 4:18pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/11 "2022-09-05T16:18:34Z")

</div>

Hmm yes that rules out the only explanation I could think of for this behaviour. Weird. I'll ask around.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 5, 2022, 4:29pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/12 "2022-09-05T16:29:58Z")

</div>

> [@ddolcimascolo](#):
>
> ```auto
> "index" : {
> "routing" : {
> "allocation" : {
> "include" : {
> "_tier_preference" : "data_content"
> }
> }
> },
> 
> ```

I do not see the point in having allocation filtering for a 3 nodes cluster where all nodes have all roles. Maybe it would be worthwhile removing it to see if that makes any difference? Could there be a bug related to `_tier_preference`?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 5, 2022, 6:10pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/13 "2022-09-05T18:10:23Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> I do not see the point in having allocation filtering for a 3 nodes cluster where all nodes have all roles

Yeah, especially considering that we don't use data tiers. But those are system indices created by ES automatically, and the problem is present on all system indices. We don't use `auto_expand_replicas` on our data indices, so I can't confirm, but I suppose the problem would be the same...

David

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 5, 2022, 7:59pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/14 "2022-09-05T19:59:28Z")

</div>

Stepping in the deep end....

What happens if you just create a new / test simple enrich processor no settings etc do you see the same behavior? like simple example from [here](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/match-enrich-policy-type.html)

Does the same failed allocation happen?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 6, 2022, 7:32am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/15 "2022-09-06T07:32:18Z")

</div>

Hi @stephenb

Thx for stepping in 🙂  
Yeah the issue is reproduced on every new enrich policy we execute, the newly created index is not replicated. This is true for all ES-managed indices and a few other one with `auto_expand_replicas`. See in the `_cat/indices` below all thoses `rep=0`...

```auto
GET /_cat/indices?v&h=index,status,pri,rep&s=index

index status pri rep
.apm-agent-configuration open 1 0
.apm-custom-link open 1 0
.async-search open 1 0
.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662448069927 open 1 0
.enrich-storcloud-brocade-switches-fe01f628717c51698c7888f56b71773b-1662448069599 open 1 0
.enrich-storcloud-ciscomds-ports-d654dbfa0a02cf00b4c5cfc2f703e0ca-1662448158314 open 1 0
.enrich-storcloud-ciscomds-switches-edcd3cb9a0db98820b3d225dfe1ffd7f-1662448158000 open 1 0
.enrich-storcloud-commvault-associations-3e235e1c535da83df878fb68cb5fc91d-1662448052374 open 1 0
.enrich-storcloud-commvault-clients-1991192cdaba2d7518b35affc0d4afbb-1662448054802 open 1 0
.enrich-storcloud-commvault-commcells-1c17d66bf2caccdb1e7837064a9291b5-1662448067469 open 1 0
.enrich-storcloud-commvault-mailboxes-a85ed3760a10feae45f942768b277dcf-1662448064148 open 1 0
.enrich-storcloud-commvault-office365-058c1165bc90ae67c13471899da6ffd6-1662448067143 open 1 0
.enrich-storcloud-commvault-storage-8bbd05bee7bd60eeb756eb16bf3a3dee-1662448067783 open 1 0
.enrich-storcloud-commvault-storage-disks-1e20a0ec38067d27d07289a8d16538ef-1662448068154 open 1 0
.enrich-storcloud-commvault-storage-libraries-45c52aeddb2edfa169e6a91c56ce34b7-1662448068550 open 1 0
.enrich-storcloud-commvault-storage-policies-744462733d98b331eef93a7469827840-1662448069195 open 1 0
.enrich-storcloud-eseries-interfaces-e563aaa537928a98c7d952a0a79c4b15-1662448157382 open 1 0
.enrich-storcloud-eseries-pools-4c08f9e40a778dcac9a3c528970a6e7d-1662448157696 open 1 0
.enrich-storcloud-eseries-systems-37bdb0d45c5ef532713df9dccc42242d-1662448157041 open 1 0
.enrich-storcloud-idrac-hardware-412df18bda3ea1469c75424f1105b8fb-1662448155727 open 1 0
.enrich-storcloud-netapp-aggregates-4148dca2273ee694523a67b72fceca86-1662447960468 open 1 0
.enrich-storcloud-netapp-clusters-589065fec3a2b65767eccf9df6a275b1-1662447961023 open 1 0
.enrich-storcloud-netapp-disks-a9470fb61b7416ded5b51d352ba2049b-1662447961387 open 1 0
.enrich-storcloud-netapp-fcp_ports-7537f138d272f914d591cc20a450bedf-1662447967880 open 1 0
.enrich-storcloud-netapp-flash-devices-b658308a5ece545739f8eefb25ac3b94-1662447968407 open 1 0
.enrich-storcloud-netapp-igroups-ded1d6989443ee7de3d45d3886df1c44-1662447968756 open 1 0
.enrich-storcloud-netapp-interfaces-3c0dc6e230681f4b415898a2ff07fe88-1662447969310 open 1 0
.enrich-storcloud-netapp-licenses-56fd9503e7421f80189581a3878efc1d-1662447970456 open 1 0
.enrich-storcloud-netapp-luns-ee42107ae37dd964192b892dbac17013-1662447971105 open 1 0
.enrich-storcloud-netapp-nodes-2200259ceb2354d90f3109e33c02073f-1662447975900 open 1 0
.enrich-storcloud-netapp-ports-c973f508fa57cd30e350ef1d2ec8208d-1662447976318 open 1 0
.enrich-storcloud-netapp-portsets-98e9458ebaf104243eb0e88eaac6cfa6-1662447977453 open 1 0
.enrich-storcloud-netapp-service-processors-f3f8d8b5a3c9cab67528b0a298e97801-1662447977883 open 1 0
.enrich-storcloud-netapp-shelves-677b0ff15b570383d62aee460699c894-1662447978270 open 1 0
.enrich-storcloud-netapp-snapmirrors-a712f2fdb35e376b3726338f5d50ec30-1662447978895 open 1 0
.enrich-storcloud-netapp-storage-switches-53b24051a6be48b0f921fb17c5f5d01c-1662447980047 open 1 0
.enrich-storcloud-netapp-volumes-f97c364b5d4ee78395cb164dc5cc1781-1662447938512 open 1 0
.enrich-storcloud-netapp-vservers-c46d81da2186e24c7506687858edf53e-1662447980380 open 1 0
.enrich-storcloud-netappgrid-buckets-999bc26cfd2e61ef36d0575525f4cfcd-1662448051852 open 1 0
.enrich-storcloud-netappgrid-datacenters-4f67f0f2f06343601f94e8eed845e84e-1662448050653 open 1 0
.enrich-storcloud-netappgrid-grids-805dd44011448f49195ed8300b949160-1662448050271 open 1 0
.enrich-storcloud-netappgrid-nodes-6e5236d21b96607bb83ba43a053b683d-1662448051028 open 1 0
.enrich-storcloud-netappgrid-tenants-d0179fa75731d562cf458fb2e9627efe-1662448051429 open 1 0
.enrich-storcloud-purestorage-arrays-6efb5a53216f570deb0af04b200efc35-1662448072723 open 1 0
.enrich-storcloud-purestorage-hostgroups-0f9b407188d5b8228c1b173bbfa6e128-1662448073066 open 1 0
.enrich-storcloud-purestorage-hosts-e4706f443fb9dfd7351d1900c774bac3-1662448074249 open 1 0
.enrich-storcloud-purestorage-pods-658ebd4faa027ffa5b386c129448a2aa-1662448075854 open 1 0
.enrich-storcloud-purestorage-protectiongroups-68dc3d5b7d9babef58025043943af9c8-1662448076183 open 1 0
.enrich-storcloud-purestorage-snapshots-ba288eecfaf1fc933e52c34f17734b4f-1662448076543 open 1 0
.enrich-storcloud-purestorage-volumes-21012d5a9eb79ca5f0efe60fc95e3c64-1662448125050 open 1 0
.enrich-storcloud-snmp-disks-c5384e2dfc5e3689dee7c0f1b6a830d2-1662448156716 open 1 0
.enrich-storcloud-snmp-hosts-624f4b4f1ed4dabb1539a013891e164e-1662448156053 open 1 0
.enrich-storcloud-snmp-interfaces-c5384e2dfc5e3689dee7c0f1b6a830d2-1662448156371 open 1 0
.enrich-storcloud-stordata-collectors-0cc0345f324f8f73e4e7f551b0d85ebe-1662448158661 open 1 0
.enrich-storcloud-stordata-toolboxes-1d2b4ee78c81a10bbcbf7dac9c570eca-1662448159599 open 1 0
.enrich-storcloud-ucs-blades-05cc1bf1ed0650a49956ce6d2f1c863e-1662448070964 open 1 0
.enrich-storcloud-ucs-chassis-1e757789d9c24dc488283c46c7960feb-1662448072040 open 1 0
.enrich-storcloud-ucs-domains-bfd7d60dcccb45806b156d6a3e828e92-1662448072419 open 1 0
.enrich-storcloud-ucs-fabric-interconnects-ec5b56dc0e77c01798bca8583801f327-1662448070583 open 1 0
.enrich-storcloud-vmware-clusters-c1c9af6b46baf0c4630138cda64b221e-1662447981278 open 1 0
.enrich-storcloud-vmware-datacenters-d67f0f7b8123894b9e3badf841b76518-1662447980946 open 1 0
.enrich-storcloud-vmware-datastores-91ac514f2f3255d9d7985f3ee873ab83-1662447981789 open 1 0
.enrich-storcloud-vmware-hosts-85493a1c4debd5f480ff7082265858a0-1662447984880 open 1 0
.enrich-storcloud-vmware-vms-blade-1662447903105 open 1 0
.enrich-storcloud-vmware-vms-c59e6a64dc2f0e2dfaa98d16a3d5cdc7-1662447990162 open 1 0
.enrich-storcloud-vmware-vms-lun-1662447900035 open 1 0
.enrich-tags-vms-to-luns-1662447904122 open 1 0
.geoip_databases open 1 0
.kibana_5 open 1 0
.kibana_7.13.3_001 open 1 0
.kibana_7.17.1_001 open 1 0
.kibana_task_manager_1 open 1 0
.kibana_task_manager_2 open 1 0
.kibana_task_manager_3 open 1 0
.kibana_task_manager_4 open 1 0
.kibana_task_manager_7.13.3_001 open 1 0
.kibana_task_manager_7.17.1_001 open 1 0
.monitoring-es-7-mb-2022.08.31 open 1 0
.monitoring-es-7-mb-2022.09.01 open 1 0
.monitoring-es-7-mb-2022.09.02 open 1 0
.monitoring-es-7-mb-2022.09.03 open 1 0
.monitoring-es-7-mb-2022.09.04 open 1 0
.monitoring-es-7-mb-2022.09.05 open 1 0
.monitoring-es-7-mb-2022.09.06 open 1 0
.monitoring-kibana-7-mb-2022.08.31 open 1 0
.monitoring-kibana-7-mb-2022.09.01 open 1 0
.monitoring-kibana-7-mb-2022.09.02 open 1 0
.monitoring-kibana-7-mb-2022.09.03 open 1 0
.monitoring-kibana-7-mb-2022.09.04 open 1 0
.monitoring-kibana-7-mb-2022.09.05 open 1 0
.monitoring-kibana-7-mb-2022.09.06 open 1 0
.slm-history-1-000006 open 1 1
.tasks-reindexed-for-8 open 1 0
.transform-internal-007 open 1 0
apm-7.17.1-error-000011 open 1 1
apm-7.17.1-error-000012 open 1 1
apm-7.17.1-metric-000011 open 1 1
apm-7.17.1-metric-000012 open 1 1
apm-7.17.1-onboarding-2022.03.24 open 1 1
apm-7.17.1-profile-000011 open 1 1
apm-7.17.1-profile-000012 open 1 1
apm-7.17.1-span-000011 open 1 1
apm-7.17.1-span-000012 open 1 1
apm-7.17.1-transaction-000011 open 1 1
apm-7.17.1-transaction-000012 open 1 1

```

Hopefully finding the root cause will replicate all of these at once !

Cheers,  
David

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 6, 2022, 9:34am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/16 "2022-09-06T09:34:26Z")

</div>

Is it only affecting enrich indices? What happens if you just create a regular index with the same settings?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 6, 2022, 10:39am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/17 "2022-09-06T10:39:11Z")

</div>

It's affecting all indices using `auto_expand_replicas` so on my cluster only system indices as we don't use this setting. I've verified it also reproduces on a regular index as per the requests below

```auto
PUT /testing_auto_expand_replicas
{
  "settings": {
    "auto_expand_replicas": "0-all"
  }
}

```

```auto
GET /testing_auto_expand_replicas/_settings
{
  "testing_auto_expand_replicas" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "provided_name" : "testing_auto_expand_replicas",
        "max_inner_result_window" : "10000",
        "creation_date" : "1662460614028",
        "number_of_replicas" : "0",
        "uuid" : "id1nwkWgQCqRCZtzpvgDWw",
        "version" : {
          "created" : "7170199"
        }
      }
    }
  }
}

```

David

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 6, 2022, 10:50am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/18 "2022-09-06T10:50:56Z")

</div>

> [@ddolcimascolo](#):
>
> ```auto
> "routing" : {
> "allocation" : {
> "include" : {
> "_tier_preference" : "data_content"
> }
> }
> }
> 
> ```

Can you also create a test index without the routing allocation setting?

---

<div class="post-metadata">

**Author:** ![ddolcimascolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddolcimascolo/32/110406_2.png) [@ddolcimascolo](https://discuss.elastic.co/u/ddolcimascolo)\
**Post date:** [September 6, 2022, 11:04am UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/19 "2022-09-06T11:04:53Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Can you also create a test index without the routing allocation setting?

Not sure how to do it. This is something added by ES automatically. Is this something unexpected?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 6, 2022, 12:27pm UTC](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569/20 "2022-09-06T12:27:03Z")

</div>

It is one of the new default settings for indices if you do not have a template that override this.

You can remove it directly from Kibana, just go to Stack Management \> Index Management, click on the index and in the Edit Settings tab delete this line:

```auto
"index.routing.allocation.include._tier_preference": "data_content",

```

[Next page](https://discuss.elastic.co/t/auto-expand-replicas-not-working-on-enrich-index/313569.md?page=2)