Hi
my iLM is stuck on awaiting allocation.
my cluster has 2x warm nodes
the warning is
Action status
Waiting for [2] shards to be allocated to nodes matching the given filters
as you can see, nothing is rolling over to the warm nodes -
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 23.2mb 319.9gb 320gb 0 10.42.255.57 10.42.255.57 instance-0000000002
278 23.9gb 25.4gb 34.5gb 60gb 42 10.42.255.14 10.42.255.14 instance-0000000000
0 0b 22.8mb 319.9gb 320gb 0 10.42.1.172 10.42.1.172 instance-0000000003
278 23.8gb 24.9gb 35gb 60gb 41 10.42.255.129 10.42.255.129 instance-0000000001
allocation explain shows no shards awaiting allocation
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
},
"status" : 400
}
this is my cat nodes
instance0002/0003 are the warm data nodes
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.42.255.57 66 99 3 0.21 0.92 1.00 rw - instance-0000000002
10.42.1.172 36 93 13 4.68 3.93 3.73 rw - instance-0000000003
10.42.255.129 73 98 12 2.60 1.82 1.61 himrst * instance-0000000001
10.42.0.120 40 99 14 2.04 2.53 2.78 mv - tiebreaker-0000000004
10.42.255.14 69 94 28 5.01 4.18 3.90 himrst - instance-0000000000
this is my ilm
"filebeat-error" : {
"version" : 2,
"modified_date" : "2021-05-04T10:31:56.095Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "30gb",
"max_age" : "1d"
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "7d",
"actions" : {
"delete" : {
"delete_searchable_snapshot" : true
}
}
},
"warm" : {
"min_age" : "1d",
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"data" : "warm"
}
},
"forcemerge" : {
"max_num_segments" : 1
},
"set_priority" : {
"priority" : 50
}
}
}
}
}
},
this is a cat nodes explain on instance 0002 and i can see it has the role
> {
> "_nodes" : {
> "total" : 1,
> "successful" : 1,
> "failed" : 0
> },
> "cluster_name" : "c5faf0690c6b477c94b63390b53bf946",
> "nodes" : {
> "xiwVv_MsThC7P4EKIBa-gQ" : {
> "name" : "instance-0000000002",
> "transport_address" : "10.42.255.57:19594",
> "host" : "10.42.255.57",
> "ip" : "10.42.255.57",
> "version" : "7.12.1",
> "build_flavor" : "default",
> "build_type" : "docker",
> "build_hash" : "3186837139b9c6b6d23c3200870651f10d3343b7",
> "total_indexing_buffer" : 107374182,
> "roles" : [
> "data_warm",
> "remote_cluster_client"
> ],
> "attributes" : {
> "logical_availability_zone" : "zone-0",
> "server_name" : "instance-0000000002.c5faf0690c6b477c94b63390b53bf946",
> "availability_zone" : "us-central1-a",
> "xpack.installed" : "true",
> "data" : "warm",
> "instance_configuration" : "gcp.data.highstorage.1",
> "transform.node" : "false",
> "region" : "unknown-region"
> },
not entirely sure why my ilm is not rolling over, does anyone have any ideas?