Thank you very much @jessgarson. I apologise for this thread being a repeat. I hope my diagnosis provides steps for the next person (as opposed to the generic page, which is helpful, too.)
The following outputs are from Postman (you could use Kibana too)
- ILM status is working:
query
https://IP:9200/_ilm/status
response
WORKING
Here is the policy and its implication for the index.
input
https://IP:9200/cowrie-logstash-2021.12.30-000018/_ilm/explain
output
{
"indices": {
"cowrie-logstash-2021.12.30-000018": {
"index": "cowrie-logstash-2021.12.30-000018",
"managed": true,
"policy": "logstash-policy",
"index_creation_date_millis": 1640884225151,
"time_since_index_creation": "878.68d",
"lifecycle_date_millis": 1643743227492,
"age": "845.59d",
"phase": "hot",
"phase_time_millis": 1643743203480,
"action": "complete",
"action_time_millis": 1643743240097,
"step": "complete",
"step_time_millis": 1643743240097,
"phase_execution": {
"policy": "logstash-policy",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "30d",
"min_docs": 1,
"max_primary_shard_docs": 200000000,
"max_primary_shard_size": "50gb"
}
}
},
"version": 4,
"modified_date_in_millis": 1622398401153
}
}
}
}
When checking for index settings, I found that ILM was listed as "completed"
Input:
https://IP:9200/cowrie-logstash-2021.12.30-000018/_settings
Output
{
"cowrie-logstash-2021.12.30-000018": {
"settings": {
"index": {
"lifecycle": {
"name": "logstash-policy",
"rollover_alias": "cowrie-logstash",
"indexing_complete": "true"
},
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"refresh_interval": "5s",
"number_of_shards": "1",
"provided_name": "<cowrie-logstash-{now/d}-000018>",
"creation_date": "1640884225151",
"number_of_replicas": "1",
"uuid": "g5GyH1aFTpC9n8pEGA0dLg",
"version": {
"created": "7160299"
}
}
}
}
}
I followed a crude approach of "reapplying" ILM settings based on this thread. I did this via Kibana by removing the ILM template to the index and applying it again.
However, the index is as is, and no tasks are pending on the cluster.
{
"cluster_name": "data_analytics_1",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 2,
"active_primary_shards": 1191,
"active_shards": 2382,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
I will continue diagnosing and updating the thread. However, from the data provided, can anyone spot an error? Please do let me know.
PS:
Since the index does not have ILM errors, the retry call does not work.