Auto_expand_replicas not working on enrich index

Elastic Stack Elasticsearch
1

Hi guys,

Our enrich indices are not replicated on our production cluster, and I don't understand why... Can you help to troubleshoot the issue?

Context

  • ES 7.17.1
  • 3 Nodes
  • Many various data indices
  • Many ingest pipelines using enrich processors
  • Many enrich policies

Nodes

GET /_cat/nodes?v
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
-           50         100  10    4.01    3.55     3.61 cdfhilmrstw *      es-03.stordata.dc
-           57          98   7    4.11    3.91     4.01 cdfhilmrstw -      es-01.stordata.dc
-           49          94   7    5.34    4.42     4.09 cdfhilmrstw -      es-02.stordata.dc

Cluster settings

GET _cluster/settings
{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "cluster_concurrent_rebalance" : "16",
          "node_concurrent_recoveries" : "4",
          "disk" : {
            "watermark" : {
              "low" : "240gb",
              "flood_stage" : "80gb",
              "high" : "120gb"
            }
          },
          "node_initial_primaries_recoveries" : "8"
        }
      },
      "max_shards_per_node" : "3000"
    },
    "indices" : {
      "lifecycle" : {
        "poll_interval" : "1m"
      },
      "recovery" : {
        "max_bytes_per_sec" : "480mb"
      }
    },
    "search" : {
      "max_buckets" : "100000"
    },
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "false"
          }
        },
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

Example index where auto_expand_replicas is not working

GET /.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d/_settings
#! this request accesses system indices: [.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167], but in a future major version, direct access to system indices will be prevented by default
{
  ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "refresh_interval" : "-1",
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662134874167",
        "max_inner_result_window" : "10000",
        "creation_date" : "1662134874167",
        "number_of_replicas" : "0",
        "uuid" : "mjvooklZSxS2b5YWKxX9Pw",
        "version" : {
          "created" : "7170199"
        },
        "warmer" : {
          "enabled" : "false"
        }
      }
    }
  }
}

Trying to manually update number_of_replicas immediately turn it back to 0.
The problem is also there on other system indices like .kibana, .monitoring, .slm-history, etc. and seems like a general problem on the cluster...

Side note: is there some logging I could enable to further track down the decision to not replicate the shards?

Thanks in advance,
David

2

Are all nodes set to data_content?

I believe auto_expand_replicas should override this, but have you checked if changing it makes any difference?

3

We don't use data tiers at all, so nodes are setup with node.roles: [] and I believe that they get all roles in this case.

Yep, I mentioned this in the original post. Setting it immediately revert it back to 0, probably in line with the decision to assign 0 replicas...

4

Hi everyone,

Is anybody able to help ?

Cheers,
David

5

node.roles: [] means the node has no roles. That'd explain it I think, this index requires a data_content node so if you have no such nodes then it will auto-expand to zero replicas.

6

Yep sorry. I mean nodes don't have the node.roles attribute at all, thus having all roles. You can confirm that by the _cat/nodes I copied in my original post

7

Hmm ok could you run the cluster allocation explain API on this shard and share the full response here?

8

So the index has a single primary shard, I ran the API on it

GET _cluster/allocation/explain
{
  "index": ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662386868413",
  "shard": 0,
  "primary": true
}

{
  "index" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662386868413",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "PS1WfPGWS4KvFDJbACEG4A",
    "name" : "es-01.stordata.dc",
    "transport_address" : "10.254.249.10:9300",
    "attributes" : {
      "ml.machine_memory" : "135075225600",
      "ml.max_open_jobs" : "512",
      "xpack.installed" : "true",
      "ml.max_jvm_size" : "33285996544",
      "transform.node" : "true"
    },
    "weight_ranking" : 1
  },
  "can_remain_on_current_node" : "yes",
  "can_rebalance_cluster" : "yes",
  "can_rebalance_to_other_node" : "no",
  "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
  "node_allocation_decisions" : [
    {
      "node_id" : "NeLqOXVAQa-38GN1jWyn_Q",
      "node_name" : "es-03.stordata.dc",
      "transport_address" : "10.254.249.12:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075368960",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1
    },
    {
      "node_id" : "qZMbs-d_QcOGh3Ve2DS1ug",
      "node_name" : "es-02.stordata.dc",
      "transport_address" : "10.254.249.11:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075360768",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1
    }
  ]
}
9

Weird, that says it should be able to expand to all 3 nodes. Could you try again with GET _cluster/allocation/explain?include_yes_decisions to get a bit more detail?

10

Sure, thx for helping out btw !

{
  "index" : ".enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662390468940",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "PS1WfPGWS4KvFDJbACEG4A",
    "name" : "es-01.stordata.dc",
    "transport_address" : "10.254.249.10:9300",
    "attributes" : {
      "ml.machine_memory" : "135075225600",
      "ml.max_open_jobs" : "512",
      "xpack.installed" : "true",
      "ml.max_jvm_size" : "33285996544",
      "transform.node" : "true"
    },
    "weight_ranking" : 1
  },
  "can_remain_on_current_node" : "yes",
  "can_rebalance_cluster" : "yes",
  "can_rebalance_to_other_node" : "no",
  "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
  "node_allocation_decisions" : [
    {
      "node_id" : "NeLqOXVAQa-38GN1jWyn_Q",
      "node_name" : "es-03.stordata.dc",
      "transport_address" : "10.254.249.12:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075368960",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "YES",
          "explanation" : "shard has no previous failures"
        },
        {
          "decider" : "replica_after_primary_active",
          "decision" : "YES",
          "explanation" : "shard is primary and can be allocated"
        },
        {
          "decider" : "enable",
          "decision" : "YES",
          "explanation" : "all allocations are allowed"
        },
        {
          "decider" : "node_version",
          "decision" : "YES",
          "explanation" : "can relocate primary shard from a node with version [7.17.1] to a node with equal-or-newer version [7.17.1]"
        },
        {
          "decider" : "snapshot_in_progress",
          "decision" : "YES",
          "explanation" : "no snapshots are currently running"
        },
        {
          "decider" : "restore_in_progress",
          "decision" : "YES",
          "explanation" : "ignored as shard is not being recovered from a snapshot"
        },
        {
          "decider" : "node_shutdown",
          "decision" : "YES",
          "explanation" : "this node is not currently shutting down"
        },
        {
          "decider" : "node_replacement",
          "decision" : "YES",
          "explanation" : "neither the source nor target node are part of an ongoing node replacement (no replacements)"
        },
        {
          "decider" : "filter",
          "decision" : "YES",
          "explanation" : "node passes include/exclude/require filters"
        },
        {
          "decider" : "same_shard",
          "decision" : "YES",
          "explanation" : "this node does not hold a copy of this shard"
        },
        {
          "decider" : "disk_threshold",
          "decision" : "YES",
          "explanation" : "enough disk for shard on node, free: [1.4tb], shard size: [207kb], free after allocating shard: [1.4tb]"
        },
        {
          "decider" : "throttling",
          "decision" : "YES",
          "explanation" : "below shard recovery limit of outgoing: [0 < 4] incoming: [0 < 4]"
        },
        {
          "decider" : "shards_limit",
          "decision" : "YES",
          "explanation" : "total shard limits are disabled: [index: -1, cluster: -1] <= 0"
        },
        {
          "decider" : "awareness",
          "decision" : "YES",
          "explanation" : "allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it"
        },
        {
          "decider" : "data_tier",
          "decision" : "YES",
          "explanation" : "index has a preference for tiers [data_content] and node has tier [data_content]"
        },
        {
          "decider" : "ccr_primary_follower",
          "decision" : "YES",
          "explanation" : "shard is not a follower and is not under the purview of this decider"
        },
        {
          "decider" : "searchable_snapshots",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshot_repository_exists",
          "decision" : "YES",
          "explanation" : "this decider only applies to indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshots_enable",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "dedicated_frozen_node",
          "decision" : "YES",
          "explanation" : "this node's data roles are not exactly [data_frozen] so it is not a dedicated frozen node"
        }
      ]
    },
    {
      "node_id" : "qZMbs-d_QcOGh3Ve2DS1ug",
      "node_name" : "es-02.stordata.dc",
      "transport_address" : "10.254.249.11:9300",
      "node_attributes" : {
        "ml.machine_memory" : "135075360768",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "33285996544",
        "transform.node" : "true"
      },
      "node_decision" : "worse_balance",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "YES",
          "explanation" : "shard has no previous failures"
        },
        {
          "decider" : "replica_after_primary_active",
          "decision" : "YES",
          "explanation" : "shard is primary and can be allocated"
        },
        {
          "decider" : "enable",
          "decision" : "YES",
          "explanation" : "all allocations are allowed"
        },
        {
          "decider" : "node_version",
          "decision" : "YES",
          "explanation" : "can relocate primary shard from a node with version [7.17.1] to a node with equal-or-newer version [7.17.1]"
        },
        {
          "decider" : "snapshot_in_progress",
          "decision" : "YES",
          "explanation" : "no snapshots are currently running"
        },
        {
          "decider" : "restore_in_progress",
          "decision" : "YES",
          "explanation" : "ignored as shard is not being recovered from a snapshot"
        },
        {
          "decider" : "node_shutdown",
          "decision" : "YES",
          "explanation" : "node [qZMbs-d_QcOGh3Ve2DS1ug] is preparing to restart, but will remain in the cluster"
        },
        {
          "decider" : "node_replacement",
          "decision" : "YES",
          "explanation" : "neither the source nor target node are part of an ongoing node replacement (no replacements)"
        },
        {
          "decider" : "filter",
          "decision" : "YES",
          "explanation" : "node passes include/exclude/require filters"
        },
        {
          "decider" : "same_shard",
          "decision" : "YES",
          "explanation" : "this node does not hold a copy of this shard"
        },
        {
          "decider" : "disk_threshold",
          "decision" : "YES",
          "explanation" : "enough disk for shard on node, free: [1.4tb], shard size: [207kb], free after allocating shard: [1.4tb]"
        },
        {
          "decider" : "throttling",
          "decision" : "YES",
          "explanation" : "below shard recovery limit of outgoing: [0 < 4] incoming: [0 < 4]"
        },
        {
          "decider" : "shards_limit",
          "decision" : "YES",
          "explanation" : "total shard limits are disabled: [index: -1, cluster: -1] <= 0"
        },
        {
          "decider" : "awareness",
          "decision" : "YES",
          "explanation" : "allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it"
        },
        {
          "decider" : "data_tier",
          "decision" : "YES",
          "explanation" : "index has a preference for tiers [data_content] and node has tier [data_content]"
        },
        {
          "decider" : "ccr_primary_follower",
          "decision" : "YES",
          "explanation" : "shard is not a follower and is not under the purview of this decider"
        },
        {
          "decider" : "searchable_snapshots",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshot_repository_exists",
          "decision" : "YES",
          "explanation" : "this decider only applies to indices backed by searchable snapshots"
        },
        {
          "decider" : "searchable_snapshots_enable",
          "decision" : "YES",
          "explanation" : "decider only applicable for indices backed by searchable snapshots"
        },
        {
          "decider" : "dedicated_frozen_node",
          "decision" : "YES",
          "explanation" : "this node's data roles are not exactly [data_frozen] so it is not a dedicated frozen node"
        }
      ]
    }
  ]
}
11

Hmm yes that rules out the only explanation I could think of for this behaviour. Weird. I'll ask around.

12

I do not see the point in having allocation filtering for a 3 nodes cluster where all nodes have all roles. Maybe it would be worthwhile removing it to see if that makes any difference? Could there be a bug related to _tier_preference?

1 Like
13

Yeah, especially considering that we don't use data tiers. But those are system indices created by ES automatically, and the problem is present on all system indices. We don't use auto_expand_replicas on our data indices, so I can't confirm, but I suppose the problem would be the same...

David

14

Stepping in the deep end....

What happens if you just create a new / test simple enrich processor no settings etc do you see the same behavior? like simple example from here

Does the same failed allocation happen?

15

Hi @stephenb

Thx for stepping in :slight_smile:
Yeah the issue is reproduced on every new enrich policy we execute, the newly created index is not replicated. This is true for all ES-managed indices and a few other one with auto_expand_replicas. See in the _cat/indices below all thoses rep=0...

GET /_cat/indices?v&h=index,status,pri,rep&s=index

index                                                                                         status pri rep
.apm-agent-configuration                                                                      open     1   0
.apm-custom-link                                                                              open     1   0
.async-search                                                                                 open     1   0
.enrich-storcloud-brocade-ports-ff6a81e1e2bdab5dce626c49b59e0e2d-1662448069927                open     1   0
.enrich-storcloud-brocade-switches-fe01f628717c51698c7888f56b71773b-1662448069599             open     1   0
.enrich-storcloud-ciscomds-ports-d654dbfa0a02cf00b4c5cfc2f703e0ca-1662448158314               open     1   0
.enrich-storcloud-ciscomds-switches-edcd3cb9a0db98820b3d225dfe1ffd7f-1662448158000            open     1   0
.enrich-storcloud-commvault-associations-3e235e1c535da83df878fb68cb5fc91d-1662448052374       open     1   0
.enrich-storcloud-commvault-clients-1991192cdaba2d7518b35affc0d4afbb-1662448054802            open     1   0
.enrich-storcloud-commvault-commcells-1c17d66bf2caccdb1e7837064a9291b5-1662448067469          open     1   0
.enrich-storcloud-commvault-mailboxes-a85ed3760a10feae45f942768b277dcf-1662448064148          open     1   0
.enrich-storcloud-commvault-office365-058c1165bc90ae67c13471899da6ffd6-1662448067143          open     1   0
.enrich-storcloud-commvault-storage-8bbd05bee7bd60eeb756eb16bf3a3dee-1662448067783            open     1   0
.enrich-storcloud-commvault-storage-disks-1e20a0ec38067d27d07289a8d16538ef-1662448068154      open     1   0
.enrich-storcloud-commvault-storage-libraries-45c52aeddb2edfa169e6a91c56ce34b7-1662448068550  open     1   0
.enrich-storcloud-commvault-storage-policies-744462733d98b331eef93a7469827840-1662448069195   open     1   0
.enrich-storcloud-eseries-interfaces-e563aaa537928a98c7d952a0a79c4b15-1662448157382           open     1   0
.enrich-storcloud-eseries-pools-4c08f9e40a778dcac9a3c528970a6e7d-1662448157696                open     1   0
.enrich-storcloud-eseries-systems-37bdb0d45c5ef532713df9dccc42242d-1662448157041              open     1   0
.enrich-storcloud-idrac-hardware-412df18bda3ea1469c75424f1105b8fb-1662448155727               open     1   0
.enrich-storcloud-netapp-aggregates-4148dca2273ee694523a67b72fceca86-1662447960468            open     1   0
.enrich-storcloud-netapp-clusters-589065fec3a2b65767eccf9df6a275b1-1662447961023              open     1   0
.enrich-storcloud-netapp-disks-a9470fb61b7416ded5b51d352ba2049b-1662447961387                 open     1   0
.enrich-storcloud-netapp-fcp_ports-7537f138d272f914d591cc20a450bedf-1662447967880             open     1   0
.enrich-storcloud-netapp-flash-devices-b658308a5ece545739f8eefb25ac3b94-1662447968407         open     1   0
.enrich-storcloud-netapp-igroups-ded1d6989443ee7de3d45d3886df1c44-1662447968756               open     1   0
.enrich-storcloud-netapp-interfaces-3c0dc6e230681f4b415898a2ff07fe88-1662447969310            open     1   0
.enrich-storcloud-netapp-licenses-56fd9503e7421f80189581a3878efc1d-1662447970456              open     1   0
.enrich-storcloud-netapp-luns-ee42107ae37dd964192b892dbac17013-1662447971105                  open     1   0
.enrich-storcloud-netapp-nodes-2200259ceb2354d90f3109e33c02073f-1662447975900                 open     1   0
.enrich-storcloud-netapp-ports-c973f508fa57cd30e350ef1d2ec8208d-1662447976318                 open     1   0
.enrich-storcloud-netapp-portsets-98e9458ebaf104243eb0e88eaac6cfa6-1662447977453              open     1   0
.enrich-storcloud-netapp-service-processors-f3f8d8b5a3c9cab67528b0a298e97801-1662447977883    open     1   0
.enrich-storcloud-netapp-shelves-677b0ff15b570383d62aee460699c894-1662447978270               open     1   0
.enrich-storcloud-netapp-snapmirrors-a712f2fdb35e376b3726338f5d50ec30-1662447978895           open     1   0
.enrich-storcloud-netapp-storage-switches-53b24051a6be48b0f921fb17c5f5d01c-1662447980047      open     1   0
.enrich-storcloud-netapp-volumes-f97c364b5d4ee78395cb164dc5cc1781-1662447938512               open     1   0
.enrich-storcloud-netapp-vservers-c46d81da2186e24c7506687858edf53e-1662447980380              open     1   0
.enrich-storcloud-netappgrid-buckets-999bc26cfd2e61ef36d0575525f4cfcd-1662448051852           open     1   0
.enrich-storcloud-netappgrid-datacenters-4f67f0f2f06343601f94e8eed845e84e-1662448050653       open     1   0
.enrich-storcloud-netappgrid-grids-805dd44011448f49195ed8300b949160-1662448050271             open     1   0
.enrich-storcloud-netappgrid-nodes-6e5236d21b96607bb83ba43a053b683d-1662448051028             open     1   0
.enrich-storcloud-netappgrid-tenants-d0179fa75731d562cf458fb2e9627efe-1662448051429           open     1   0
.enrich-storcloud-purestorage-arrays-6efb5a53216f570deb0af04b200efc35-1662448072723           open     1   0
.enrich-storcloud-purestorage-hostgroups-0f9b407188d5b8228c1b173bbfa6e128-1662448073066       open     1   0
.enrich-storcloud-purestorage-hosts-e4706f443fb9dfd7351d1900c774bac3-1662448074249            open     1   0
.enrich-storcloud-purestorage-pods-658ebd4faa027ffa5b386c129448a2aa-1662448075854             open     1   0
.enrich-storcloud-purestorage-protectiongroups-68dc3d5b7d9babef58025043943af9c8-1662448076183 open     1   0
.enrich-storcloud-purestorage-snapshots-ba288eecfaf1fc933e52c34f17734b4f-1662448076543        open     1   0
.enrich-storcloud-purestorage-volumes-21012d5a9eb79ca5f0efe60fc95e3c64-1662448125050          open     1   0
.enrich-storcloud-snmp-disks-c5384e2dfc5e3689dee7c0f1b6a830d2-1662448156716                   open     1   0
.enrich-storcloud-snmp-hosts-624f4b4f1ed4dabb1539a013891e164e-1662448156053                   open     1   0
.enrich-storcloud-snmp-interfaces-c5384e2dfc5e3689dee7c0f1b6a830d2-1662448156371              open     1   0
.enrich-storcloud-stordata-collectors-0cc0345f324f8f73e4e7f551b0d85ebe-1662448158661          open     1   0
.enrich-storcloud-stordata-toolboxes-1d2b4ee78c81a10bbcbf7dac9c570eca-1662448159599           open     1   0
.enrich-storcloud-ucs-blades-05cc1bf1ed0650a49956ce6d2f1c863e-1662448070964                   open     1   0
.enrich-storcloud-ucs-chassis-1e757789d9c24dc488283c46c7960feb-1662448072040                  open     1   0
.enrich-storcloud-ucs-domains-bfd7d60dcccb45806b156d6a3e828e92-1662448072419                  open     1   0
.enrich-storcloud-ucs-fabric-interconnects-ec5b56dc0e77c01798bca8583801f327-1662448070583     open     1   0
.enrich-storcloud-vmware-clusters-c1c9af6b46baf0c4630138cda64b221e-1662447981278              open     1   0
.enrich-storcloud-vmware-datacenters-d67f0f7b8123894b9e3badf841b76518-1662447980946           open     1   0
.enrich-storcloud-vmware-datastores-91ac514f2f3255d9d7985f3ee873ab83-1662447981789            open     1   0
.enrich-storcloud-vmware-hosts-85493a1c4debd5f480ff7082265858a0-1662447984880                 open     1   0
.enrich-storcloud-vmware-vms-blade-1662447903105                                              open     1   0
.enrich-storcloud-vmware-vms-c59e6a64dc2f0e2dfaa98d16a3d5cdc7-1662447990162                   open     1   0
.enrich-storcloud-vmware-vms-lun-1662447900035                                                open     1   0
.enrich-tags-vms-to-luns-1662447904122                                                        open     1   0
.geoip_databases                                                                              open     1   0
.kibana_5                                                                                     open     1   0
.kibana_7.13.3_001                                                                            open     1   0
.kibana_7.17.1_001                                                                            open     1   0
.kibana_task_manager_1                                                                        open     1   0
.kibana_task_manager_2                                                                        open     1   0
.kibana_task_manager_3                                                                        open     1   0
.kibana_task_manager_4                                                                        open     1   0
.kibana_task_manager_7.13.3_001                                                               open     1   0
.kibana_task_manager_7.17.1_001                                                               open     1   0
.monitoring-es-7-mb-2022.08.31                                                                open     1   0
.monitoring-es-7-mb-2022.09.01                                                                open     1   0
.monitoring-es-7-mb-2022.09.02                                                                open     1   0
.monitoring-es-7-mb-2022.09.03                                                                open     1   0
.monitoring-es-7-mb-2022.09.04                                                                open     1   0
.monitoring-es-7-mb-2022.09.05                                                                open     1   0
.monitoring-es-7-mb-2022.09.06                                                                open     1   0
.monitoring-kibana-7-mb-2022.08.31                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.01                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.02                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.03                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.04                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.05                                                            open     1   0
.monitoring-kibana-7-mb-2022.09.06                                                            open     1   0
.slm-history-1-000006                                                                         open     1   1
.tasks-reindexed-for-8                                                                        open     1   0
.transform-internal-007                                                                       open     1   0
apm-7.17.1-error-000011                                                                       open     1   1
apm-7.17.1-error-000012                                                                       open     1   1
apm-7.17.1-metric-000011                                                                      open     1   1
apm-7.17.1-metric-000012                                                                      open     1   1
apm-7.17.1-onboarding-2022.03.24                                                              open     1   1
apm-7.17.1-profile-000011                                                                     open     1   1
apm-7.17.1-profile-000012                                                                     open     1   1
apm-7.17.1-span-000011                                                                        open     1   1
apm-7.17.1-span-000012                                                                        open     1   1
apm-7.17.1-transaction-000011                                                                 open     1   1
apm-7.17.1-transaction-000012                                                                 open     1   1

Hopefully finding the root cause will replicate all of these at once !

Cheers,
David

16

Is it only affecting enrich indices? What happens if you just create a regular index with the same settings?

17

It's affecting all indices using auto_expand_replicas so on my cluster only system indices as we don't use this setting. I've verified it also reproduces on a regular index as per the requests below

PUT /testing_auto_expand_replicas
{
  "settings": {
    "auto_expand_replicas": "0-all"
  }
}

GET /testing_auto_expand_replicas/_settings
{
  "testing_auto_expand_replicas" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "provided_name" : "testing_auto_expand_replicas",
        "max_inner_result_window" : "10000",
        "creation_date" : "1662460614028",
        "number_of_replicas" : "0",
        "uuid" : "id1nwkWgQCqRCZtzpvgDWw",
        "version" : {
          "created" : "7170199"
        }
      }
    }
  }
}

David

18

Can you also create a test index without the routing allocation setting?

1 Like
19

Not sure how to do it. This is something added by ES automatically. Is this something unexpected?

20

It is one of the new default settings for indices if you do not have a template that override this.

You can remove it directly from Kibana, just go to Stack Management > Index Management, click on the index and in the Edit Settings tab delete this line:

"index.routing.allocation.include._tier_preference": "data_content",