# Index rolling window of data **URL:** \ **Category:** Elasticsearch\ **Created:** [June 21, 2012, 11:15pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187 "2012-06-21T23:15:10Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ashish\_Nigam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_nigam/32/2804_2.png) [@Ashish\_Nigam](https://discuss.elastic.co/u/Ashish_Nigam)\ **Post date:** [June 21, 2012, 11:15pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/1 "2012-06-21T23:15:10Z")
HI, I need help/suggestions in defining the right indexing/data migration strategy. I have to create tenant based index storage. My use case covers around 100 tenants. I need to allow user to search data for last 30 days. So that would mean that I would need to create a rolling indexes of 30 days. My estimate is that there will be around 4 million rows per tenant in 30 days. I am thinking of creating an index per tenant per day and maintain an alias for 30 days. But that would mean 100 \* 30 = 3000 active indexes at any point of time. Is this a huge number of indexes compared to amount of data that needs to be managed? Another way can be to maintain just one index per tenant and move out a day worth of data every day so that active alias always have 30 days worth of data. Please suggest me the right strategy to move forward here. If second option is better, I don't know the right way to selectively move a day's worth of data from an index to some another place. Thanks Ashish ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 22, 2012, 8:19pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/2 "2012-06-22T20:19:16Z")
Hi, With so few docs/tenant/30 days I'd use just 1 index/tenant, add all content to it, and remove docs \> 30 days old from it on a nightly basis. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, June 21, 2012 7:15:10 PM UTC-4, Ashish Nigam wrote: > HI, > I need help/suggestions in defining the right indexing/data migration > strategy. > I have to create tenant based index storage. My use case covers around 100 > tenants. > I need to allow user to search data for last 30 days. So that would mean > that I would need to create a rolling indexes of 30 days. My estimate is > that there will be around 4 million rows per tenant in 30 days. > > I am thinking of creating an index per tenant per day and maintain an > alias for 30 days. But that would mean 100 \* 30 = 3000 active indexes at > any point of time. Is this a huge number of indexes compared to amount of > data that needs to be managed? > > Another way can be to maintain just one index per tenant and move out a > day worth of data every day so that active alias always have 30 days worth > of data. > > Please suggest me the right strategy to move forward here. If second > option is better, I don't know the right way to selectively move a day's > worth of data from an index to some another place. > > Thanks > Ashish ---
**Author:** ![Ashish\_Nigam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_nigam/32/2804_2.png) [@Ashish\_Nigam](https://discuss.elastic.co/u/Ashish_Nigam)\ **Post date:** [June 22, 2012, 9:20pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/3 "2012-06-22T21:20:24Z")
Thanks Otis. I would stick to one index/tenant. To remove docs on nightly basis from the index, is there any good way to perform soft delete, i.e move a day worth of data from an index to another index? I can probably search all the data, move it to another index and then delete it from original index. But if there is any other more efficient way that doesn't explicitly involve a client executing these three steps, that might be more efficient. On Jun 22, 2012, at 1:19 PM, Otis Gospodnetic wrote: > Hi, > > With so few docs/tenant/30 days I'd use just 1 index/tenant, add all content to it, and remove docs \> 30 days old from it on a nightly basis. > > ## Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Thursday, June 21, 2012 7:15:10 PM UTC-4, Ashish Nigam wrote: > HI, > I need help/suggestions in defining the right indexing/data migration strategy. > I have to create tenant based index storage. My use case covers around 100 tenants. > I need to allow user to search data for last 30 days. So that would mean that I would need to create a rolling indexes of 30 days. My estimate is that there will be around 4 million rows per tenant in 30 days. > > I am thinking of creating an index per tenant per day and maintain an alias for 30 days. But that would mean 100 \* 30 = 3000 active indexes at any point of time. Is this a huge number of indexes compared to amount of data that needs to be managed? > > Another way can be to maintain just one index per tenant and move out a day worth of data every day so that active alias always have 30 days worth of data. > > Please suggest me the right strategy to move forward here. If second option is better, I don't know the right way to selectively move a day's worth of data from an index to some another place. > > Thanks > Ashish ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 23, 2012, 4:01am UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/4 "2012-06-23T04:01:29Z")
I don't think there is something built-in that does this. But it shouldn't be too hard to write an app that queries your main index, gets all to-be-deleted docs and uses their \_source to index them to another index. Might be a nice addition to ES or a standalone tool, as a matter of fact. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Friday, June 22, 2012 5:20:24 PM UTC-4, Ashish Nigam wrote: > Thanks Otis. > I would stick to one index/tenant. > To remove docs on nightly basis from the index, is there any good way to > perform soft delete, i.e move a day worth of data from an index to another > index? > I can probably search all the data, move it to another index and then > delete it from original index. But if there is any other more efficient way > that doesn't explicitly involve a client executing these three steps, that > might be more efficient. > > On Jun 22, 2012, at 1:19 PM, Otis Gospodnetic wrote: > > Hi, > > With so few docs/tenant/30 days I'd use just 1 index/tenant, add all > content to it, and remove docs \> 30 days old from it on a nightly basis. > > ## Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Thursday, June 21, 2012 7:15:10 PM UTC-4, Ashish Nigam wrote: > > > HI, > > I need help/suggestions in defining the right indexing/data migration > > strategy. > > I have to create tenant based index storage. My use case covers around > > 100 tenants. > > I need to allow user to search data for last 30 days. So that would mean > > that I would need to create a rolling indexes of 30 days. My estimate is > > that there will be around 4 million rows per tenant in 30 days. > > > > I am thinking of creating an index per tenant per day and maintain an > > alias for 30 days. But that would mean 100 \* 30 = 3000 active indexes at > > any point of time. Is this a huge number of indexes compared to amount of > > data that needs to be managed? > > > > Another way can be to maintain just one index per tenant and move out a > > day worth of data every day so that active alias always have 30 days worth > > of data. > > > > Please suggest me the right strategy to move forward here. If second > > option is better, I don't know the right way to selectively move a day's > > worth of data from an index to some another place. > > > > Thanks > > Ashish ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 24, 2012, 7:51am UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/5 "2012-06-24T07:51:50Z")
Hi Ashish, On Fri, Jun 22, 2012 at 11:20 PM, Ashish Nigam [ashish@skyhighnetworks.com](mailto:ashish@skyhighnetworks.com)wrote: > To remove docs on nightly basis from the index, is there any good way to > perform soft delete, i.e move a day worth of data from an index to another > index? You might want to check the TTL feature for this, so your docs get automatically deleted after 30 days. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > --Alexander ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 25, 2012, 12:14pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/6 "2012-06-25T12:14:42Z")
Note, is considerably cheaper to delete an index than delete data from an index. Having 3000 shards is possible, but will "cost" you in terms of the number of nodes you will need to start in order to support it. What I would do is do index per day, and on that index, have several shards holding data for all tenants, and using routing (the tenant "name") to direct specific tenant data to a shard. Here is the presentation I gave that explains it: [https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics](https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics). On Sun, Jun 24, 2012 at 9:51 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hi Ashish, > > On Fri, Jun 22, 2012 at 11:20 PM, Ashish Nigam \<[ashish@skyhighnetworks.com](mailto:ashish@skyhighnetworks.com) > > > wrote: > > > To remove docs on nightly basis from the index, is there any good way to > > perform soft delete, i.e move a day worth of data from an index to another > > index? > > You might want to check the TTL feature for this, so your docs get > automatically deleted after 30 days. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html) > > > > > --Alexander ---
**Author:** ![Ashish\_Nigam\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashish_nigam_2/32/2804_2.png) [@Ashish\_Nigam\_2](https://discuss.elastic.co/u/Ashish_Nigam_2)\ **Post date:** [June 25, 2012, 3:43pm UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/7 "2012-06-25T15:43:27Z")
Thanks for your feedback Shay. There can be few high traffic tenants where I would need to store as much as 20 million entries in a day. As routing works on a single shard, will it be fine to store so much data on a single shard. I am still in investigation mode and do not have a cluster to verify the performance with respect to data volume and shard allocations. If it is standard to store this much data on a single shard, I would move ahead with this assumption. On Jun 25, 2012, at 5:14 AM, Shay Banon wrote: > Note, is considerably cheaper to delete an index than delete data from an index. > > Having 3000 shards is possible, but will "cost" you in terms of the number of nodes you will need to start in order to support it. What I would do is do index per day, and on that index, have several shards holding data for all tenants, and using routing (the tenant "name") to direct specific tenant data to a shard. > > Here is the presentation I gave that explains it: [https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics](https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics). > > On Sun, Jun 24, 2012 at 9:51 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hi Ashish, > > On Fri, Jun 22, 2012 at 11:20 PM, Ashish Nigam [ashish@skyhighnetworks.com](mailto:ashish@skyhighnetworks.com) wrote: > To remove docs on nightly basis from the index, is there any good way to perform soft delete, i.e move a day worth of data from an index to another index? > > You might want to check the TTL feature for this, so your docs get automatically deleted after 30 days. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html) > > --Alexander ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:22am UTC](https://discuss.elastic.co/t/index-rolling-window-of-data/8187/8 "2017-07-06T03:22:39Z")