Data Retention for a Data Stream showing 'Disabled' - Why?

Elastic Stack Kibana
1

Hi there,

I was wonderig that the manual says this:

A value in the data retention column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored.

URL: Index management in Kibana | Elasticsearch Guide [8.18] | Elastic

But why does in my case the data retention show ‘disabled’ if ILM is turned on for this data stream? Should it not show a number there instead? I have defined a delete phase after 40 days.

image

Also interesting is the content of the online help. So are there 2 different kind of data retention definitions in place?

image
image1409×387 60.6 KB

I would appreciate some insinght.

Thanks!

2

Yeah, this is confusing, the documentation you shared is for data stream lifecycle, it is not the same thing as ILM.

If you have ILM, the retention is managed by the ILM policy, the Data Retention column you see in Kibana only shows information if the retention is managed by a Data Stream lifecycle, so if you are using ILM it will shows up as Disabled.

I'm not sure exactly what is the main difference as I found it confusing and just stuck with ILM, it seems that they do basically the same thing, but in some different way.

1 Like
3

Asking AI: In the elastic stack what is the difference between: data stream lifecycle and ILM?

The key difference between data stream lifecycle and Index Lifecycle Management (ILM) in the Elastic Stack lies in their scope, complexity, and feature set: ILM provides detailed and flexible lifecycle management for indices and data streams, while data stream lifecycle is a simpler, streamlined system specifically designed for data streams, optimized for ease of use and automation.

Data Stream Lifecycle

  • Data stream lifecycle only applies to data streams (time-series, append-only data).

  • It allows automated operations like data retention, automatic rollover, and downsampling.

  • Lifecycle configuration is applied directly to the data stream—retention can be set on individual streams or globally.

  • It does not currently support advanced lifecycle actions like data tiering, shrinking, or searchable snapshots.

  • Designed for environments (including Serverless and standard Elastic deployments) where simplicity and automation are prioritized over customization.

  • New backing indices created under the data stream are managed by the data stream lifecycle, while older indices may still be managed by ILM during migration.

Index Lifecycle Management (ILM)

  • ILM can manage both individual indices and data streams.

  • Provides fully customizable policies, including automatic rollover, data tiering (hot/warm/cold/frozen), shrinking, force merges, deletions, and more.

  • Offers granular control over how and when data moves between performance-focused and cost-focused hardware tiers.

  • More complex to configure, but powerful for advanced use cases.

  • In environments where both ILM and data stream lifecycle could apply, ILM typically takes precedence unless specified otherwise.

  • ILM is not available in Elasticsearch Serverless; only data stream lifecycle is supported there.

Side-by-Side Comparison

Feature Data Stream Lifecycle ILM
Scope Data streams only Indices & data streams
Configuration Streamlined, easy Highly customizable
Advanced actions Limited (no tiering, shrink, etc.) Supports tiering, shrink, merge, delete
Deployment types Most deployments + Serverless Versioned clusters, not Serverless
Use cases Simplified retention, rollover Complex data lifecycles, detailed policies

Data stream lifecycle is recommended for straightforward management of data streams, especially in Serverless environments or where ease of use is preferred, while ILM excels in scenarios requiring detailed, fine-tuned lifecycle policies for indices or data streams.