ILM Waiting for allocation on warm nodes

Hi

my iLM is stuck on awaiting allocation.

my cluster has 2x warm nodes

the warning is

Action status

Waiting for [2] shards to be allocated to nodes matching the given filters

as you can see, nothing is rolling over to the warm nodes -

shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
     0           0b    23.2mb    319.9gb      320gb            0 10.42.255.57  10.42.255.57  instance-0000000002
   278       23.9gb    25.4gb     34.5gb       60gb           42 10.42.255.14  10.42.255.14  instance-0000000000
     0           0b    22.8mb    319.9gb      320gb            0 10.42.1.172   10.42.1.172   instance-0000000003
   278       23.8gb    24.9gb       35gb       60gb           41 10.42.255.129 10.42.255.129 instance-0000000001

allocation explain shows no shards awaiting allocation

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}

this is my cat nodes

instance0002/0003 are the warm data nodes

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.42.255.57            66          99   3    0.21    0.92     1.00 rw        -      instance-0000000002
10.42.1.172             36          93  13    4.68    3.93     3.73 rw        -      instance-0000000003
10.42.255.129           73          98  12    2.60    1.82     1.61 himrst    *      instance-0000000001
10.42.0.120             40          99  14    2.04    2.53     2.78 mv        -      tiebreaker-0000000004
10.42.255.14            69          94  28    5.01    4.18     3.90 himrst    -      instance-0000000000

this is my ilm

  "filebeat-error" : {
    "version" : 2,
    "modified_date" : "2021-05-04T10:31:56.095Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "30gb",
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "7d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        },
        "warm" : {
          "min_age" : "1d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            }
          }
        }
      }
    }
  },

this is a cat nodes explain on instance 0002 and i can see it has the role

> {
>   "_nodes" : {
>     "total" : 1,
>     "successful" : 1,
>     "failed" : 0
>   },
>   "cluster_name" : "c5faf0690c6b477c94b63390b53bf946",
>   "nodes" : {
>     "xiwVv_MsThC7P4EKIBa-gQ" : {
>       "name" : "instance-0000000002",
>       "transport_address" : "10.42.255.57:19594",
>       "host" : "10.42.255.57",
>       "ip" : "10.42.255.57",
>       "version" : "7.12.1",
>       "build_flavor" : "default",
>       "build_type" : "docker",
>       "build_hash" : "3186837139b9c6b6d23c3200870651f10d3343b7",
>       "total_indexing_buffer" : 107374182,
>       "roles" : [
>         "data_warm",
>         "remote_cluster_client"
>       ],
>       "attributes" : {
>         "logical_availability_zone" : "zone-0",
>         "server_name" : "instance-0000000002.c5faf0690c6b477c94b63390b53bf946",
>         "availability_zone" : "us-central1-a",
>         "xpack.installed" : "true",
>         "data" : "warm",
>         "instance_configuration" : "gcp.data.highstorage.1",
>         "transform.node" : "false",
>         "region" : "unknown-region"
>       },

not entirely sure why my ilm is not rolling over, does anyone have any ideas?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.