# How to fix retention log and how to manage data retention time in Elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323>\
**Category:** Kibana\
**Tags:** ilm-index-lifecycle-management\
**Created:** [January 31, 2023, 12:12pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323 "2023-01-31T12:12:25Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mosa](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@Mosa](https://discuss.elastic.co/u/Mosa)\
**Post date:** [January 31, 2023, 12:12pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/1 "2023-01-31T12:12:25Z")

</div>

Hi Team,

I've configured my retention log in my Elasticsearch and I hope managing my data retention time. But no thing is changed after 3 days in my configuration just to test it. And no data is deleted too after 3 days. And worst of all, the data size keeps increasing day by day even though I have already set it 30GB. It exceeds that.

Below is that I've did:

First, I created Index Lifecycle Policies and I've specified the order in the Hot phase, Warm phase, Cold phase and the last Delete phase.

Second, I created my Index Management and I connected it with Index lifecycle management.

So if there is someone here can help me to fix it.

Thanks for your feedback.

---

<div class="post-metadata">

**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\
**Post date:** [January 31, 2023, 12:32pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/2 "2023-01-31T12:32:26Z")

</div>

Hello,

Can you post the settings from the API, of the index you are trying to use ILM on? Can you also post the configuration of your ILM policy?

---

<div class="post-metadata">

**Author:** ![Mosa](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@Mosa](https://discuss.elastic.co/u/Mosa)\
**Post date:** [January 31, 2023, 1:08pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/3 "2023-01-31T13:08:11Z")

</div>

@mverbeek

Thanks for your feedback.  
Below is the Logstash of index management's settings:

 ![ILM Policy-Logstash](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fbbce11c9fe40947bfc04f0daa110e74aabb207e.png)

```auto
{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "logstash-policy",
        "rollover_alias": ""
      },
      "routing": {
        "allocation": {
          "include": {
            "_tier_preference": "data_content"
          }
        }
      },
      "number_of_shards": "1",
      "provided_name": "logstash",
      "creation_date": "1673351562470",
      "number_of_replicas": "1",
      "uuid": "rMbSCwWIQE6ElOgyGBE1OA",
      "version": {
        "created": "7170399"
      }
    }
  },
  "defaults": {
    "index": {
      "flush_after_merge": "512mb",
      "final_pipeline": "_none",
      "max_inner_result_window": "100",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "1m"
        }
      },
      "max_terms_count": "65536",
      "rollup": {
        "source": {
          "name": "",
          "uuid": ""
        }
      },
      "lifecycle": {
        "parse_origination_date": "false",
        "step": {
          "wait_time_threshold": "12h"
        },
        "indexing_complete": "false",
        "origination_date": "-1"
      },
      "routing_partition_size": "1",
      "force_memory_term_dictionary": "false",
      "max_docvalue_fields_search": "100",
      "merge": {
        "scheduler": {
          "max_thread_count": "1",
          "auto_throttle": "true",
          "max_merge_count": "6"
        },
        "policy": {
          "floor_segment": "2mb",
          "max_merge_at_once_explicit": "30",
          "max_merge_at_once": "10",
          "max_merged_segment": "5gb",
          "expunge_deletes_allowed": "10.0",
          "segments_per_tier": "10.0",
          "deletes_pct_allowed": "33.0"
        }
      },
      "max_refresh_listeners": "1000",
      "max_regex_length": "1000",
      "load_fixed_bitset_filters_eagerly": "true",
      "number_of_routing_shards": "1",
      "write": {
        "wait_for_active_shards": "1"
      },
      "verified_before_close": "false",
      "mapping": {
        "coerce": "false",
        "nested_fields": {
          "limit": "50"
        },
        "depth": {
          "limit": "20"
        },
        "field_name_length": {
          "limit": "9223372036854775807"
        },
        "total_fields": {
          "limit": "1000"
        },
        "nested_objects": {
          "limit": "10000"
        },
        "ignore_malformed": "false",
        "dimension_fields": {
          "limit": "16"
        }
      },
      "source_only": "false",
      "soft_deletes": {
        "enabled": "true",
        "retention": {
          "operations": "0"
        },
        "retention_lease": {
          "period": "12h"
        }
      },
      "max_script_fields": "32",
      "query": {
        "default_field": [
          "*"
        ],
        "parse": {
          "allow_unmapped_fields": "true"
        }
      },
      "format": "0",
      "frozen": "false",
      "sort": {
        "missing": [],
        "mode": [],
        "field": [],
        "order": []
      },
      "priority": "1",
      "codec": "default",
      "max_rescore_window": "10000",
      "max_adjacency_matrix_filters": "100",
      "analyze": {
        "max_token_count": "10000"
      },
      "gc_deletes": "60s",
      "top_metrics_max_size": "10",
      "optimize_auto_generated_id": "true",
      "max_ngram_diff": "1",
      "hidden": "false",
      "translog": {
        "generation_threshold_size": "64mb",
        "flush_threshold_size": "512mb",
        "sync_interval": "5s",
        "retention": {
          "size": "-1",
          "age": "-1"
        },
        "durability": "REQUEST"
      },
      "auto_expand_replicas": "false",
      "mapper": {
        "dynamic": "true"
      },
      "recovery": {
        "type": ""
      },
      "requests": {
        "cache": {
          "enable": "true"
        }
      },
      "data_path": "",
      "highlight": {
        "max_analyzed_offset": "1000000"
      },
      "routing": {
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "include": {
            "_tier": ""
          },
          "disk": {
            "watermark": {
              "ignore": "false"
            }
          },
          "exclude": {
            "_tier": ""
          },
          "require": {
            "_tier": ""
          },
          "enable": "all",
          "total_shards_per_node": "-1"
        }
      },
      "search": {
        "slowlog": {
          "level": "TRACE",
          "threshold": {
            "fetch": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            },
            "query": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          }
        },
        "idle": {
          "after": "30s"
        },
        "throttled": "false"
      },
      "fielddata": {
        "cache": "node"
      },
      "default_pipeline": "_none",
      "max_slices_per_scroll": "1024",
      "shard": {
        "check_on_startup": "false"
      },
      "xpack": {
        "watcher": {
          "template": {
            "version": ""
          }
        },
        "version": "",
        "ccr": {
          "following_index": "false"
        }
      },
      "percolator": {
        "map_unmapped_fields_as_text": "false"
      },
      "allocation": {
        "max_retries": "5",
        "existing_shards_allocator": "gateway_allocator"
      },
      "refresh_interval": "1s",
      "indexing": {
        "slowlog": {
          "reformat": "true",
          "threshold": {
            "index": {
              "warn": "-1",
              "trace": "-1",
              "debug": "-1",
              "info": "-1"
            }
          },
          "source": "1000",
          "level": "TRACE"
        }
      },
      "compound_format": "0.1",
      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },
      "max_result_window": "10000",
      "store": {
        "stats_refresh_interval": "10s",
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": [],
        "snapshot": {
          "snapshot_name": "",
          "index_uuid": "",
          "cache": {
            "prewarm": {
              "enabled": "true"
            },
            "enabled": "true",
            "excluded_file_types": []
          },
          "repository_uuid": "",
          "uncached_chunk_size": "-1b",
          "index_name": "",
          "partial": "false",
          "blob_cache": {
            "metadata_files": {
              "max_length": "64kb"
            }
          },
          "repository_name": "",
          "snapshot_uuid": ""
        }
      },
      "queries": {
        "cache": {
          "enabled": "true"
        }
      },
      "shard_limit": {
        "group": "normal"
      },
      "warmer": {
        "enabled": "true"
      },
      "max_shingle_diff": "3",
      "query_string": {
        "lenient": "false"
      }
    }
  }
}

```

And this is my ILM policy

 ![ILM Policy](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78ac8536c548f3898235b59971c4090e47dc54f2.png)

 ![Hot phase](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e65afd682443e91ece45bb3511b24af26a94d48.png)

 ![Warm phase](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b15cbd26bb6180c77797b1edad884eaecfc87811.png)

 ![Cold phase](https://us1.discourse-cdn.com/elastic/original/3X/8/2/828d10af14d9148ed874d59efca75e51ce56bbcb.png)

 ![Delete phase](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76ee1fd8257dbfbd374a37405093a006424e932c.png)

---

<div class="post-metadata">

**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\
**Post date:** [January 31, 2023, 1:56pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/4 "2023-01-31T13:56:19Z")

</div>

My first guess would be the missing roll-over alias.

See here a guide on how to do it: [Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html)

With

```auto
GET <target>/_ilm/explain

```

You should get more information.

---

<div class="post-metadata">

**Author:** ![Mosa](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@Mosa](https://discuss.elastic.co/u/Mosa)\
**Post date:** [January 31, 2023, 2:06pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/5 "2023-01-31T14:06:24Z")

</div>

Thanks a lot.

I'll try it and I'll come back here later to give you a feedback what is happened.

Thank you.

---

<div class="post-metadata">

**Author:** ![Mosa](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@Mosa](https://discuss.elastic.co/u/Mosa)\
**Post date:** [February 2, 2023, 1:32pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/6 "2023-02-02T13:32:10Z")

</div>

Thank you.  
My retention is working now.

---

<div class="post-metadata">

**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\
**Post date:** [February 2, 2023, 1:44pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/7 "2023-02-02T13:44:29Z")

</div>

Glad to hear that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2023, 1:45pm UTC](https://discuss.elastic.co/t/how-to-fix-retention-log-and-how-to-manage-data-retention-time-in-elasticsearch/324323/8 "2023-03-02T13:45:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.