# Selective retnetion for an index **URL:** \ **Category:** Elasticsearch\ **Created:** [April 3, 2025, 6:27am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720 "2025-04-03T06:27:17Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Anusha\_Kusanghi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusha_kusanghi/32/98109_2.png) [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Post date:** [April 3, 2025, 6:27am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/1 "2025-04-03T06:27:17Z")
We have a requirement in Elasticsearch where, for a single index, we need to store certain fields for one month and others for three months. Since there is no direct selective retention with ILM , we aim to achieve this by using the clone option. The plan is to send specific fields to `index_one`, which has a longer ILM policy, and the remaining fields to `index_two`, which has a shorter ILM policy. Both indices can then be viewed collectively under `index*` in Kibana. Is this a good approach? Additionally, how resource-intensive is the cloning process when dealing with large datasets? Parllely we also tried with custom scripts that run on the index and delete the field but this will have additional load on our elastic cluster as it parses each record for field deletion so had to rule this out. We are also exploring Tranfsorm Indices. Is there any other way to have selective retention for fields in a single index to save storage. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 3, 2025, 7:10am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/2 "2025-04-03T07:10:02Z")
I'm wondering if you can use [field level security](https://www.elastic.co/guide/en/elasticsearch/reference/current/field-level-security.html) to change the visibility of a field for some users after a month. Not sure but may be a trick that would avoid expensive reindex operations. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 3, 2025, 7:21am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/3 "2025-04-03T07:21:11Z")
> [@Anusha\_Kusanghi](#): > > The plan is to send specific fields to `index_one`, which has a longer ILM policy, and the remaining fields to `index_two`, which has a shorter ILM policy. Both indices can then be viewed collectively under `index*` in Kibana. Although you can send specific fields to different indices, be aware that Elasticsearch does not support joins so you will not be able to run queries across all fields. Deleting fields will require reindexing, which can be expensive. An option to save some space and avoid expensive reindexing might be to simply have two series of time-based indices. Into the first one you send the documents with all the fields and you retain this for just 1 month. Into a separate index you send the stripped down documents and keep these around for 3 months. This means that you duplicate storage for the reduced set of fields for one month out of three, but it saves a lot of potentially expensive processing and is simple and low risk. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 3, 2025, 7:33am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/4 "2025-04-03T07:33:45Z")
Very good idea @Christian_Dahlqvist! 😉 BTW joins are coming in 8.18 with ES|QL. ---
**Author:** ![Anusha\_Kusanghi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusha_kusanghi/32/98109_2.png) [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Post date:** [April 4, 2025, 5:39am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/5 "2025-04-04T05:39:28Z")
How you replicate same data to indieces ? As I discribed above my ideas is to use clone and duplicate data ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2025, 5:54am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/6 "2025-04-04T05:54:42Z")
You index the data twice, e.g. using Logstash with a [clone](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html) processor and dual outputs. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2025, 6:18am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/7 "2025-04-04T06:18:53Z")
> [@dadoonet](#): > > BTW joins are coming in 8.18 with ES|QL. Is that general joins where you can join multiple large indices or more like a lookup where you can join a large index towards a limited size data set? ---
**Author:** ![Anusha\_Kusanghi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusha_kusanghi/32/98109_2.png) [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Post date:** [April 7, 2025, 4:54am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/8 "2025-04-07T04:54:55Z")
My data is not always an array to apply split filter and index into 2 different outputs ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 7, 2025, 5:55am UTC](https://discuss.elastic.co/t/selective-retnetion-for-an-index/376720/9 "2025-04-07T05:55:09Z")
I meant to write clone processor but somehow got it wrong. I have fixed my previous post and provided a link to the appropriate section in the docs.