Index lifecycle policy does not apply to newly created timeseries indexes

I'm using version 7.8.1 of the elastic stack for indexing log files with filebeat into daily logstash-yyyy-mm-dd indexes. I've created an index pattern and template for the indexes and a Logstash_Index_Retention policy and applied it to the template. All I am concerned with is deleting old indexes after a certain number of days so in the policy I disable rollover in the hot phase and activate the delete phase with a number of days from index creation. In order to apply the policy to existing indexes, I issue the following command:

curl -X PUT "localhost:9200/logstash*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "Logstash_Index_Retention"
    }
  }
}
'

This works and I can see that the policy applies to all existing indexes. But I notice after a few dayst that the policy is not applied to new logstash-* indexes created later. What am I doing wrong?
Thanks.

I still need help with this. I've tried different things, but nothing works.. As far as I can tell this webinar implies that what I have done should work. It seems like this should be a simple thing to do, but I haven't found much help with it in the searches I've done. Is this even supposed to work?
Here's more information:
The lifecycle policy:

PUT _ilm/policy/Logstash_Index_Retention
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "10d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The index template settings:

{
  "index": {
    "lifecycle": {
      "name": "Logstash_Index_Retention"
    },
    "number_of_shards": "1",
    "refresh_interval": "5s"
  }
}

The index pattern is

logstash-*

You can try the ILM lifecycle explain API and see if it is applied for your indices. Also, please share the actual settings of one of those indices, as the index template only is applied when a new index is created.

Thanks for your reply. The explain API shows that the policy is applied to all current indexes because I've done so manually. Each day I check it i see that newer indexes don't have the policy applied. Anyway here are the settings for today's index, but I've already applied the policy to it:

{
  "settings": {
    "index": {
      "mapping": {
        "ignore_malformed": "true"
      },
      "refresh_interval": "30s",
      "translog": {
        "sync_interval": "1m"
      },
      "provided_name": "logstash-2021-07-12",
      "query": {
        "default_field": "message"
      },
      "creation_date": "1626047968911",
      "analysis": {
        "normalizer": {
          "lowercase_normalizer": {
            "filter": [
              "lowercase"
            ],
            "type": "custom"
          }
        },
        "analyzer": {
          "log_analyzer": {
            "lowercase": "true",
            "pattern": "\\W+",
            "type": "pattern"
          },
          "list_analyzer": {
            "pattern": "[\\s/,;:|]+",
            "type": "pattern"
          }
        }
      },
      "priority": "100",
      "number_of_replicas": "1",
      "uuid": "JfGKJxzcQHidYGO94ENNyw",
      "version": {
        "created": "7080199"
      },
      "lifecycle": {
        "name": "Logstash_Index_Retention",
        "rollover_alias": ""
      },
      "number_of_shards": "2"
    }
  },
  "defaults": {
    "index": {
      "flush_after_merge": "512mb",
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "force_memory_id_terms_dictionary": "false",
      "lifecycle": {
        "parse_origination_date": "false",
        "indexing_complete": "false",
        "origination_date": "-1"
      },
      "routing_partition_size": "1",
      "force_memory_term_dictionary": "false",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "1",
          "auto_throttle": "true",
          "max_merge_count": "6"
        },
        "policy": {
          "reclaim_deletes_weight": "2.0",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "verified_before_close": "false",
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "false",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "analyze": {
        "max_token_count": "10000"
      },
      "gc_deletes": "60s",
      "top_metrics_max_size": "10",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "hidden": "false",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "retention": {
          "size": "512MB",
          "age": "12h"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "1000000"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "idle": {
          "after": "30s"
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false"
      },
      "allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      },
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": []
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

Here's the settings for today's index in another cluster with the same configuration where the policy hasn't been automatically applied:

{
  "settings": {
    "index": {
      "mapping": {
        "ignore_malformed": "true"
      },
      "refresh_interval": "30s",
      "number_of_shards": "2",
      "translog": {
        "sync_interval": "1m"
      },
      "provided_name": "logstash-2021-07-12",
      "query": {
        "default_field": "message"
      },
      "creation_date": "1626047986994",
      "analysis": {
        "normalizer": {
          "lowercase_normalizer": {
            "filter": [
              "lowercase"
            ],
            "type": "custom"
          }
        },
        "analyzer": {
          "log_analyzer": {
            "lowercase": "true",
            "pattern": "\\W+",
            "type": "pattern"
          },
          "list_analyzer": {
            "pattern": "[\\s/,;:|]+",
            "type": "pattern"
          }
        }
      },
      "number_of_replicas": "1",
      "uuid": "J2IikUXUSrC-LcaT9Ynpjw",
      "version": {
        "created": "7080199"
      }
    }
  },
  "defaults": {
    "index": {
      "flush_after_merge": "512mb",
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "force_memory_id_terms_dictionary": "false",
      "lifecycle": {
        "name": "",
        "parse_origination_date": "false",
        "indexing_complete": "false",
        "rollover_alias": "",
        "origination_date": "-1"
      },
      "routing_partition_size": "1",
      "force_memory_term_dictionary": "false",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "1",
          "auto_throttle": "true",
          "max_merge_count": "6"
        },
        "policy": {
          "reclaim_deletes_weight": "2.0",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "verified_before_close": "false",
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "false",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "priority": "1",
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "analyze": {
        "max_token_count": "10000"
      },
      "gc_deletes": "60s",
      "top_metrics_max_size": "10",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "hidden": "false",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "retention": {
          "size": "512MB",
          "age": "12h"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "1000000"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "idle": {
          "after": "30s"
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false"
      },
      "allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      },
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": []
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

@spinscale Is there anything else you need to see? Thanks!

can you share the explain api output as well?

Here is the explain API. I applied the policy to all indexes yesterday, but you can see that today's index, July 13, does not have it ("managed" : false):

$ curl -X GET "localhost:9200/logstash-*/_ilm/explain?pretty"
{
  "indices" : {
    "logstash-2021-07-11" : {
      "index" : "logstash-2021-07-11",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1625961579390,
      "age" : "2.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109064898,
      "action" : "complete",
      "action_time_millis" : 1626109066294,
      "step" : "complete",
      "step_time_millis" : 1626109066294,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    },
    "logstash-2021-07-12" : {
      "index" : "logstash-2021-07-12",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1626047986994,
      "age" : "1.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109065873,
      "action" : "complete",
      "action_time_millis" : 1626109066691,
      "step" : "complete",
      "step_time_millis" : 1626109066691,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    },
    "logstash-2021-07-10" : {
      "index" : "logstash-2021-07-10",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1625875178425,
      "age" : "3.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109065306,
      "action" : "complete",
      "action_time_millis" : 1626109066470,
      "step" : "complete",
      "step_time_millis" : 1626109066470,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    },
    "logstash-2021-07-13" : {
      "index" : "logstash-2021-07-13",
      "managed" : false
    },
    "logstash-2021-07-08" : {
      "index" : "logstash-2021-07-08",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1625702380804,
      "age" : "5.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109065043,
      "action" : "complete",
      "action_time_millis" : 1626109066337,
      "step" : "complete",
      "step_time_millis" : 1626109066337,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    },
    "logstash-2021-07-09" : {
      "index" : "logstash-2021-07-09",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1625788775870,
      "age" : "4.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109065175,
      "action" : "complete",
      "action_time_millis" : 1626109066382,
      "step" : "complete",
      "step_time_millis" : 1626109066382,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    },
    "logstash-2021-07-07" : {
      "index" : "logstash-2021-07-07",
      "managed" : true,
      "policy" : "Logstash_Index_Retention",
      "lifecycle_date_millis" : 1625615989742,
      "age" : "6.53d",
      "phase" : "hot",
      "phase_time_millis" : 1626109065420,
      "action" : "complete",
      "action_time_millis" : 1626109066524,
      "step" : "complete",
      "step_time_millis" : 1626109066524,
      "phase_execution" : {
        "policy" : "Logstash_Index_Retention",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1623891465545
      }
    }
  }
}

Your index template mentions number of shards and refresh interval along with the ILM policy. has that been successfully applied?

Apparently not. Here are the settings for today's index. The values are different (30s and 2 instead of 5s and 1):

{
  "settings": {
    "index": {
      "mapping": {
        "ignore_malformed": "true"
      },
      "refresh_interval": "30s",
      "number_of_shards": "2",
      "translog": {
        "sync_interval": "1m"
      },
      "provided_name": "logstash-2021-07-13",
      "query": {
        "default_field": "message"
      },
      "creation_date": "1626134374574",
      "analysis": {
        "normalizer": {
          "lowercase_normalizer": {
            "filter": [
              "lowercase"
            ],
            "type": "custom"
          }
        },
        "analyzer": {
          "log_analyzer": {
            "lowercase": "true",
            "pattern": "\\W+",
            "type": "pattern"
          },
          "list_analyzer": {
            "pattern": "[\\s/,;:|]+",
            "type": "pattern"
          }
        }
      },
      "number_of_replicas": "1",
      "uuid": "sBMBhqmlQM6r6t5M-LBdhw",
      "version": {
        "created": "7080199"
      }
    }
  },
  "defaults": {
    "index": {
      "flush_after_merge": "512mb",
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "force_memory_id_terms_dictionary": "false",
      "lifecycle": {
        "name": "",
        "parse_origination_date": "false",
        "indexing_complete": "false",
        "rollover_alias": "",
        "origination_date": "-1"
      },
      "routing_partition_size": "1",
      "force_memory_term_dictionary": "false",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "1",
          "auto_throttle": "true",
          "max_merge_count": "6"
        },
        "policy": {
          "reclaim_deletes_weight": "2.0",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "verified_before_close": "false",
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "false",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "priority": "1",
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "analyze": {
        "max_token_count": "10000"
      },
      "gc_deletes": "60s",
      "top_metrics_max_size": "10",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "hidden": "false",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "retention": {
          "size": "512MB",
          "age": "12h"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "1000000"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "idle": {
          "after": "30s"
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false"
      },
      "allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      },
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": []
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

But, on an index where the policy has been applied the values are the same (30s and 2 instead of 5s and 1) though they are shown in a different place I don't know were the values in the index template come from:

{
  "settings": {
    "index": {
      "mapping": {
        "ignore_malformed": "true"
      },
      "refresh_interval": "30s",
      "translog": {
        "sync_interval": "1m"
      },
      "provided_name": "logstash-2021-07-12",
      "query": {
        "default_field": "message"
      },
      "creation_date": "1626047986994",
      "analysis": {
        "normalizer": {
          "lowercase_normalizer": {
            "filter": [
              "lowercase"
            ],
            "type": "custom"
          }
        },
        "analyzer": {
          "log_analyzer": {
            "lowercase": "true",
            "pattern": "\\W+",
            "type": "pattern"
          },
          "list_analyzer": {
            "pattern": "[\\s/,;:|]+",
            "type": "pattern"
          }
        }
      },
      "priority": "100",
      "number_of_replicas": "1",
      "uuid": "J2IikUXUSrC-LcaT9Ynpjw",
      "version": {
        "created": "7080199"
      },
      "lifecycle": {
        "name": "Logstash_Index_Retention"
      },
      "number_of_shards": "2"
    }
  },
  "defaults": {
    "index": {
      "flush_after_merge": "512mb",
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "force_memory_id_terms_dictionary": "false",
      "lifecycle": {
        "parse_origination_date": "false",
        "indexing_complete": "false",
        "rollover_alias": "",
        "origination_date": "-1"
      },
      "routing_partition_size": "1",
      "force_memory_term_dictionary": "false",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "1",
          "auto_throttle": "true",
          "max_merge_count": "6"
        },
        "policy": {
          "reclaim_deletes_weight": "2.0",
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "verified_before_close": "false",
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "false",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "analyze": {
        "max_token_count": "10000"
      },
      "gc_deletes": "60s",
      "top_metrics_max_size": "10",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "hidden": "false",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "retention": {
          "size": "512MB",
          "age": "12h"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "1000000"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "idle": {
          "after": "30s"
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false"
      },
      "allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      },
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": []
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

Here is the structure I'm using to set the logstash-* index when I install the cluster:

{
  "index_patterns": ["logstash-*"],
  "template": {
    "settings": {
      "index.mapping.ignore_malformed": true,
      "index.query.default_field": "message",
      "index.refresh_interval": "30s",
      "index.translog.sync_interval": "1m",
      "index.number_of_shards": 2,
      "index.number_of_replicas": 1,
      "index.analysis": {
        "analyzer": {
          "log_analyzer": {
            "type": "pattern",
            "pattern": "\\W+",
            "lowercase": true
          },
          "list_analyzer": {
            "type": "pattern",
            "pattern": "[\\s/,;:|]+"
          }
        },
        "normalizer": {
          "lowercase_normalizer": {
            "type": "custom",
            "filter": ["lowercase"]
          }
        }
      }
    },
    "mappings": {
      "dynamic": true,
      "dynamic_templates": [
      {
        "no_index_fields_past_depth_3": {
          "path_match": "*.*.*",
          "match_mapping_type": "object",
          "mapping": {
            "type": "object",
            "enabled": false
          }
        }
      },
      {
        "create_keyword_index_for_all_string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "normalizer": "lowercase_normalizer",
            "ignore_above": 1000
          }
        }
      }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "thread": {
          "type": "text",
          "analyzer": "log_analyzer",
          "norms": false,
          "similarity": "boolean",
          "fields": {
            "keyword": {
              "type": "keyword",
              "normalizer": "lowercase_normalizer"
            }
          }
        },
        "message": {
          "type": "text",
          "analyzer": "log_analyzer",
          "norms": false,
          "similarity": "boolean"
        },
        "geoip":{
          "dynamic":true,
          "properties":{
            "location":{
              "type":"geo_point",
              "doc_values": true
            }
          },
          "type":"object"
        },
        "tags": {
          "type": "text",
          "analyzer": "standard"
        },
        "log.file.path": {
          "type": "text",
          "analyzer": "list_analyzer"
        }
      }
    }
  },
  "priority": 500,
  "version": 1,
  "_meta": {
    "description": "logstash index template"
  }
}

probably you have other index templates overwriting this one?

No, this the only one I use. It's applied through the rest API when the cluster is installed. The settings that apply the ILM policy are created later in Kibana though along with the policy itself.

I've simplified the index template settings in Kibana to what I set initially (see my first message) to see if that has any effect. I'm not sure why the other settings were added.

{
  "index": {
    "lifecycle": {
      "name": "Logstash_Index_Retention"
    }
  }
}

Is there a way I can include this in the index settings and mapping that I apply with the rest API? Then apply the policy the same way at the same time?

Also, here are the policy settings:

PUT _ilm/policy/Logstash_Index_Retention
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "10d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Hey,

I fully trust you that this is the only template you use, my idea was more that logstash might have been automatically installed a template, that overwrites this one :slight_smile:

--Alex

Thanks! Your questions prodded me to think a bit more about how I'm going about this and I've found a good solution. Instead of trying to add the policy after installing the cluster, I'm adding it in at install time and putting the name of the policy in the index template that I use when installing. I'm seeing that this works when new indexes are created with the logstash pattern. So it must be that trying to create the policy and add it to the index template after the fact doesn't let new indexes inherit the policy for some reason. Thanks very much for your help!

1 Like