ILM policy not applied

Elastic Stack Elasticsearch
3

Thank you very much @jessgarson. I apologise for this thread being a repeat. I hope my diagnosis provides steps for the next person (as opposed to the generic page, which is helpful, too.)

The following outputs are from Postman (you could use Kibana too)

  1. ILM status is working:

query

https://IP:9200/_ilm/status

response

WORKING

Here is the policy and its implication for the index.

input

https://IP:9200/cowrie-logstash-2021.12.30-000018/_ilm/explain

output

{
    "indices": {
        "cowrie-logstash-2021.12.30-000018": {
            "index": "cowrie-logstash-2021.12.30-000018",
            "managed": true,
            "policy": "logstash-policy",
            "index_creation_date_millis": 1640884225151,
            "time_since_index_creation": "878.68d",
            "lifecycle_date_millis": 1643743227492,
            "age": "845.59d",
            "phase": "hot",
            "phase_time_millis": 1643743203480,
            "action": "complete",
            "action_time_millis": 1643743240097,
            "step": "complete",
            "step_time_millis": 1643743240097,
            "phase_execution": {
                "policy": "logstash-policy",
                "phase_definition": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_age": "30d",
                            "min_docs": 1,
                            "max_primary_shard_docs": 200000000,
                            "max_primary_shard_size": "50gb"
                        }
                    }
                },
                "version": 4,
                "modified_date_in_millis": 1622398401153
            }
        }
    }
}

When checking for index settings, I found that ILM was listed as "completed"

Input:

https://IP:9200/cowrie-logstash-2021.12.30-000018/_settings

Output

{
    "cowrie-logstash-2021.12.30-000018": {
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "logstash-policy",
                    "rollover_alias": "cowrie-logstash",
                    "indexing_complete": "true"
                },
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "refresh_interval": "5s",
                "number_of_shards": "1",
                "provided_name": "<cowrie-logstash-{now/d}-000018>",
                "creation_date": "1640884225151",
                "number_of_replicas": "1",
                "uuid": "g5GyH1aFTpC9n8pEGA0dLg",
                "version": {
                    "created": "7160299"
                }
            }
        }
    }
}

I followed a crude approach of "reapplying" ILM settings based on this thread. I did this via Kibana by removing the ILM template to the index and applying it again.

However, the index is as is, and no tasks are pending on the cluster.

{
    "cluster_name": "data_analytics_1",
    "status": "green",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 2,
    "active_primary_shards": 1191,
    "active_shards": 2382,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 0,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 100.0
}

I will continue diagnosing and updating the thread. However, from the data provided, can anyone spot an error? Please do let me know.

PS:
Since the index does not have ILM errors, the retry call does not work.