I am using ILM via Logstash to manage my indices
I have a new index that was created in 31/10 and for some unknown reason, it has become read-only, I noticed it has block.write=true on it.
My ILM policy shouldn't do this as no condition was met - The policy:
{ "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "100gb" }, "set_priority": { "priority": 100 } } }, "warm": { "min_age": "0ms", "actions": { "allocate": { "number_of_replicas": 1, "include": {}, "exclude": {}, "require": { "box_type": "warm" } }, "forcemerge": { "max_num_segments": 1 }, "set_priority": { "priority": 50 }, "shrink": { "number_of_shards": 1 } } }, "cold": { "min_age": "30d", "actions": { "freeze": {}, "set_priority": { "priority": 0 } } }, "delete": { "min_age": "365d", "actions": { "delete": {} } } } } }
I have DEBUG logs from the Master at the time it became a read-only:
[2019-11-04T12:30:04,705][DEBUG][o.e.i.IndicesService ] [Master] creating Index [[12months-retention-2019.10.31-000001/IQvq2eTrSqyX-Gd476zvLA]], shards [5]/[1] - reason [metadata verification]
[2019-11-04T12:30:04,706][DEBUG][o.e.i.m.MapperService ] [Master] [12months-retention-2019.10.31-000001] using dynamic[true]
[2019-11-04T12:30:04,707][DEBUG][o.e.i.c.b.BitsetFilterCache] [Master] [12months-retention-2019.10.31-000001] clearing all bitsets because [close]
[2019-11-04T12:30:04,707][DEBUG][o.e.i.c.q.IndexQueryCache] [Master] [12months-retention-2019.10.31-000001] full cache clear, reason [close]
[2019-11-04T12:30:04,707][DEBUG][o.e.i.c.b.BitsetFilterCache] [Master] [12months-retention-2019.10.31-000001] clearing all bitsets because [close]
[2019-11-04T12:31:48,285][DEBUG][o.e.x.i.IndexLifecycleRunner] [Master] [12months-retention-2019.10.31-000001] running periodic policy with current-step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]
This is the index template:
{
"order": 0,
"version": 60001,
"index_patterns": [
"12months-retention-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "12months_policy",
"rollover_alias": "12months-retention"
}
}
},
"mappings": {},
"aliases": {}
}
This is the current index status:
No new index was created, it did not rollover.
The Elasticsearch version is 6.8.3
Please advise, thanks.