# Facing challange during segregate the data from one index to another index **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring, ilm-index-lifecycle-management\ **Created:** [June 19, 2023, 10:45am UTC](https://discuss.elastic.co/t/facing-challange-during-segregate-the-data-from-one-index-to-another-index/336349 "2023-06-19T10:45:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [June 19, 2023, 10:45am UTC](https://discuss.elastic.co/t/facing-challange-during-segregate-the-data-from-one-index-to-another-index/336349/1 "2023-06-19T10:45:13Z")
Hello Team, We are using Akamai for CDN and WAF. We have configured datastream on akamai and getting those logs on our elasticsaerch. Please refer the below link for same: [Stream logs to Elasticsearch](https://techdocs.akamai.com/datastream2/docs/stream-elasticsearch) As per doc we have created index on akamai dashboard and we have created index pattren for same. But Akamai doesn't support `timestamp` so its not creating new index for new date. So all the data is coming into single index and its creating issue when single indices size is growing. We want to segregate the data on the basis of date and tries few things: 1. We have tried to configure the ILM: a). Created the ILM on the basis of age (1 Day) b). Created an index template and include that index in this template. c) Linked the template with ILM policy which we created. d) Created an alias for the index. e) Attach the ILM policy and alias with the index. But its doing `rollover` operation and creating new index e.g `akami-sf-00002` with size of `450byte` but all the data exist in default index which we defined during akamai setup. `Rollover` is creating new index but data can't came into this index because index name is static on akamai side. 1. Second option we have tried using `reindex`. Reindex copy the data into new index but its very slow process and we need to delete the data from source index once reindex is complete. We want to acheive to create new index automatically on the basis of date or if any funtion is available in elasticsearch so we can segregate the data on the basis of date. Can you please help me here? Thank You ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2023, 12:30pm UTC](https://discuss.elastic.co/t/facing-challange-during-segregate-the-data-from-one-index-to-another-index/336349/2 "2023-06-19T12:30:02Z")
What is the `index` that you set on the akamai side? And what does your ILM and Templates looks like? Please share them using the preformatted text option, the `` button. If you want to send the data directly to Elasticsearch you may have two approachs in this case, one is using Data Streams and following [this documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html). But I'm not sure this Akamai output would work with data streams. To configure rollover using daily indices you would need to follow this part of [the same documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams). You basically need to create the ILM policy in the way you want to rollover after 1 day for example, then you need an index template where you need to have the `index.lifecycle.name` set to your ILM policy and `index.lifecycle.rollover_alias` set to an alias, something like `akamai` for example. You will need to manually create the first indice, something like this: ```auto PUT akamai-000001 { "aliases": { "akamai": { "is_write_index": true } } } ``` Then on the Akamai side you will need to use this same alias as the index name. ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [June 20, 2023, 6:37am UTC](https://discuss.elastic.co/t/facing-challange-during-segregate-the-data-from-one-index-to-another-index/336349/3 "2023-06-20T06:37:27Z")
@leandrojmp , Thank You for your quick response. I was doing almost same steps except `rollover_alias` during index template creation. ```auto PUT _index_template/timeseries_template { "index_patterns": ["timeseries-*"], "template": { "settings": { "number_of_shards": 1, "number_of_replicas": 1, "index.lifecycle.name": "timeseries_policy", "index.lifecycle.rollover_alias": "timeseries" } } } ``` Now i have made the changes and able to get response during lifecycle check progress. ```auto GET akamai-sf-*/_ilm/explain ``` ```auto { "indices" : { "akamai-sf-01" : { "index" : "akamai-sf-01", "managed" : true, "policy" : "akamai-sf", "lifecycle_date_millis" : 1687236792712, "age" : "1.63h", "phase" : "hot", "phase_time_millis" : 1687236795050, "action" : "rollover", "action_time_millis" : 1687236796457, "step" : "check-rollover-ready", "step_time_millis" : 1687236796457, "phase_execution" : { "policy" : "akamai-sf", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_age" : "1d" }, "set_priority" : { "priority" : 100 } } }, "version" : 1, "modified_date_in_millis" : 1687236168849 } } } } ``` Now when date will change rollover should work as per our expectation. Will update you tomorrow morning. Thank You ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2023, 6:38am UTC](https://discuss.elastic.co/t/facing-challange-during-segregate-the-data-from-one-index-to-another-index/336349/4 "2023-07-18T06:38:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.