# Apply ILM Policy to existing DataStream Index **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [March 16, 2021, 8:55pm UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422 "2021-03-16T20:55:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Karl\_Falconer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_falconer/32/74795_2.png) [@Karl\_Falconer](https://discuss.elastic.co/u/Karl_Falconer)\ **Post date:** [March 16, 2021, 8:55pm UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/1 "2021-03-16T20:55:10Z")
We have an exiting datastream `foo-stream` and two backing indices `.foo-stream-000001` and `.foo-stream-000002`. Our application is writing/reading to the index `foo-stream` and the overall expectation is that ILM would automatically create and adjust the writable index (-00000#) as the policy thresholds are met. The policy: ```auto "rollover": { "max_age": "1d", "max_size": "50gb" } ``` The problem is that the index `.foo-stream-000001` is 32 days old and contains 1.1TB of data. We found that the ILM policy was not attached to this specific index, but does match the index pattern. When manually applying the ILM policy to this index, `.foo-stream-000002` was created, but it contains no documents and the `.foo-stream-000001` has stopped receiving documents. I would have expected that when applying the ILM Policy to `.foo-stream-000001` it would have split the data into at least 32 different indexes. What am I missing? The explain for `.foo-stream-000001` ```auto { "indices": { ".foo-stream-000001": { "index": ".foo-stream-000001", "managed": true, "phase_execution": { "policy": "foo-stream", "modified_date_in_millis": 1615925871750, "phase_definition": { "actions": { "rollover": { "max_age": "1d", "max_size": "50gb" }, "set_priority": { "priority": 100 } }, "min_age": "0ms" }, "version": 5 }, "action_time_millis": 1615919902092, "age": "2.22h", "step_time_millis": 1615919902092, "phase_time_millis": 1615918967960, "phase": "hot", "step": "complete", "lifecycle_date_millis": 1615919894909, "policy": "foo-stream", "action": "complete" } } } ``` ---
**Author:** ![AClerk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aclerk/32/55297_2.png) [@AClerk](https://discuss.elastic.co/u/AClerk)\ **Post date:** [March 17, 2021, 12:07am UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/2 "2021-03-17T00:07:00Z")
AFAIK The policy will work from NOW. It will not calculate what should have been in retrospect. ---
**Author:** ![Karl\_Falconer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_falconer/32/74795_2.png) [@Karl\_Falconer](https://discuss.elastic.co/u/Karl_Falconer)\ **Post date:** [March 17, 2021, 12:27am UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/3 "2021-03-17T00:27:29Z")
1. Is there a way to manually split the first index? 2. Why would the first index still be receiving new documents? The `.foo-stream-000002` index still has 0 documents, but the `.foo-stream-000001` index continues to grow ---
**Author:** ![AClerk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aclerk/32/55297_2.png) [@AClerk](https://discuss.elastic.co/u/AClerk)\ **Post date:** [March 17, 2021, 12:41am UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/4 "2021-03-17T00:41:54Z")
1. You can [re-index](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) 2. You need to apply your logic in index settings, for existing indices. Elasticsearch applies templates to new indices based on an index pattern that matches the index name. [Create or update index template API | Elasticsearch Reference [7.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html#indices-templates-v1) ---
**Author:** ![Karl\_Falconer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karl_falconer/32/74795_2.png) [@Karl\_Falconer](https://discuss.elastic.co/u/Karl_Falconer)\ **Post date:** [March 17, 2021, 3:26pm UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/5 "2021-03-17T15:26:27Z")
I've discovered a bit more information which might explain why this is not working as intended, and is a bit of a rabbit hole situation. When trying to reindex from the `.foo-stream-000001` index into the `foo-stream` to then have the ILM policy perform as expected, I get the error ```auto no write index is defined for alias [foo-stream]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index ``` ILM should have automatically updated the is\_write\_index when creating the `.foo-stream-000002` but seems like it did not. I believe the trouble there is documented in this issue: [Alias incorrectly refers to data stream · Issue #67730 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/67730) Effectively the Index template contained an alias which had the same name as the DataStream. This then allowed the backing indexes to inherit the aliases, but they should not have. So at this point it seems that both indexes are in an inconsistent state and it is unclear how to walk this back. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 14, 2021, 3:27pm UTC](https://discuss.elastic.co/t/apply-ilm-policy-to-existing-datastream-index/267422/6 "2021-04-14T15:27:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.