# Large index design question **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2012, 2:21pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454 "2012-04-24T14:21:10Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Sky\_Stebnicki1](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@Sky\_Stebnicki1](https://discuss.elastic.co/u/Sky_Stebnicki1)\ **Post date:** [April 24, 2012, 2:21pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/1 "2012-04-24T14:21:10Z")
Hi all, We've been testing elasticsearch with our application and are really enjoying impressive performance and rock-solid stability. I have a design question about what would be the most efficient way to index (1) large set of highly active date and (2) an even larger set of archived data. Basically we have tens of millions of documents but about 80% of them are in archive state and 10-20% are read/updated 95% of the time. My question is this: would it be more efficient to store the archived documents into a separate type like "/index/mydata\_arch" or to just use a filtered query to cache the results and flag the archived documents as we index them? We are working on setting up benchmarks to test this ourselves in a real-word environment but I wanted to ask the experts here too and see if you had any input. Thanks so much for your help! Sky ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 24, 2012, 3:40pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/2 "2012-04-24T15:40:37Z")
I'd consider using separate indices and aliases. Keeping the active index smaller would help with the performance. Will you know before you index which docs are archived data and which are active? Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > Hi all, > > We've been testing elasticsearch with our application and are really > enjoying impressive performance and rock-solid stability. I have a design > question about what would be the most efficient way to index (1) large set > of highly active date and (2) an even larger set of archived > data. Basically we have tens of millions of documents but about 80% of them > are in archive state and 10-20% are read/updated 95% of the time. > > My question is this: would it be more efficient to store the archived > documents into a separate type like "/index/mydata\_arch" or to just use a > filtered query to cache the results and flag the archived documents as we > index them? > > We are working on setting up benchmarks to test this ourselves in a > real-word environment but I wanted to ask the experts here too and see if > you had any input. > > Thanks so much for your help! > > Sky ---
**Author:** ![Sky\_Stebnicki1](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@Sky\_Stebnicki1](https://discuss.elastic.co/u/Sky_Stebnicki1)\ **Post date:** [April 26, 2012, 5:19pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/3 "2012-04-26T17:19:00Z")
Sorry for my late reply, for some reason I was not notified when you posted. Documents can move from active to archived at any time so we will know as the document is indexed or updated where it belongs. In our database we split the records into two separate tables for performance reasons. So your suggestion is to create a completely separate index or type? Sorry, I'm fairly new to the Elasticsearch terminology... Thanks, Sky On Tuesday, April 24, 2012 8:40:37 AM UTC-7, Berkay Mollamustafaoglu wrote: > I'd consider using separate indices and aliases. Keeping the active index > smaller would help with the performance. Will you know before you index > which docs are archived data and which are active? > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > Hi all, > > > > We've been testing elasticsearch with our application and are really > > enjoying impressive performance and rock-solid stability. I have a design > > question about what would be the most efficient way to index (1) large set > > of highly active date and (2) an even larger set of archived > > data. Basically we have tens of millions of documents but about 80% of them > > are in archive state and 10-20% are read/updated 95% of the time. > > > > My question is this: would it be more efficient to store the archived > > documents into a separate type like "/index/mydata\_arch" or to just use a > > filtered query to cache the results and flag the archived documents as we > > index them? > > > > We are working on setting up benchmarks to test this ourselves in a > > real-word environment but I wanted to ask the experts here too and see if > > you had any input. > > > > Thanks so much for your help! > > > > Sky ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 26, 2012, 5:57pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/4 "2012-04-26T17:57:50Z")
No worries. Yes I'd recommend 2 separate indices not types. This would allow you to optimize them differently. There are number of params that you can only set in index level. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Thu, Apr 26, 2012 at 1:19 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > Sorry for my late reply, for some reason I was not notified when you > posted. Documents can move from active to archived at any time so we will > know as the document is indexed or updated where it belongs. In our > database we split the records into two separate tables for performance > reasons. > > So your suggestion is to create a completely separate index or type? > Sorry, I'm fairly new to the Elasticsearch terminology... > > Thanks, > > Sky > > On Tuesday, April 24, 2012 8:40:37 AM UTC-7, Berkay Mollamustafaoglu wrote: > > > I'd consider using separate indices and aliases. Keeping the active index > > smaller would help with the performance. Will you know before you index > > which docs are archived data and which are active? > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > > > Hi all, > > > > > > We've been testing elasticsearch with our application and are really > > > enjoying impressive performance and rock-solid stability. I have a design > > > question about what would be the most efficient way to index (1) large set > > > of highly active date and (2) an even larger set of archived > > > data. Basically we have tens of millions of documents but about 80% of them > > > are in archive state and 10-20% are read/updated 95% of the time. > > > > > > My question is this: would it be more efficient to store the archived > > > documents into a separate type like "/index/mydata\_arch" or to just use a > > > filtered query to cache the results and flag the archived documents as we > > > index them? > > > > > > We are working on setting up benchmarks to test this ourselves in a > > > real-word environment but I wanted to ask the experts here too and see if > > > you had any input. > > > > > > Thanks so much for your help! > > > > > > Sky ---
**Author:** ![Sky\_Stebnicki1](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@Sky\_Stebnicki1](https://discuss.elastic.co/u/Sky_Stebnicki1)\ **Post date:** [April 26, 2012, 11:46pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/5 "2012-04-26T23:46:51Z")
What did you mean by "aliases"? If we use two separate indexes would I have to merge/union the results at the application level if we needed to search over both active and archived data? On Thursday, April 26, 2012 10:57:50 AM UTC-7, Berkay Mollamustafaoglu wrote: > No worries. Yes I'd recommend 2 separate indices not types. This would > allow you to optimize them differently. There are number of params that you > can only set in index level. > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Thu, Apr 26, 2012 at 1:19 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > Sorry for my late reply, for some reason I was not notified when you > > posted. Documents can move from active to archived at any time so we will > > know as the document is indexed or updated where it belongs. In our > > database we split the records into two separate tables for performance > > reasons. > > > > So your suggestion is to create a completely separate index or type? > > Sorry, I'm fairly new to the Elasticsearch terminology... > > > > Thanks, > > > > Sky > > > > On Tuesday, April 24, 2012 8:40:37 AM UTC-7, Berkay Mollamustafaoglu > > wrote: > > > > > I'd consider using separate indices and aliases. Keeping the active > > > index smaller would help with the performance. Will you know before you > > > index which docs are archived data and which are active? > > > > > > Regards, > > > Berkay Mollamustafaoglu > > > mberkay on yahoo, google and skype > > > > > > On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki \<[sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com) > > > > > > > wrote: > > > > > > > Hi all, > > > > > > > > We've been testing elasticsearch with our application and are really > > > > enjoying impressive performance and rock-solid stability. I have a design > > > > question about what would be the most efficient way to index (1) large set > > > > of highly active date and (2) an even larger set of archived > > > > data. Basically we have tens of millions of documents but about 80% of them > > > > are in archive state and 10-20% are read/updated 95% of the time. > > > > > > > > My question is this: would it be more efficient to store the archived > > > > documents into a separate type like "/index/mydata\_arch" or to just use a > > > > filtered query to cache the results and flag the archived documents as we > > > > index them? > > > > > > > > We are working on setting up benchmarks to test this ourselves in a > > > > real-word environment but I wanted to ask the experts here too and see if > > > > you had any input. > > > > > > > > Thanks so much for your help! > > > > > > > > Sky ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 27, 2012, 2:28am UTC](https://discuss.elastic.co/t/large-index-design-question/7454/6 "2012-04-27T02:28:46Z")
You don't have to merge the results yourself. ES allows you create an alias that references to multiple indices and then query the alias like a normal index. It'll do the work for you. They are quite powerful and useful. Details here: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Thu, Apr 26, 2012 at 7:46 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > What did you mean by "aliases"? If we use two separate indexes would I > have to merge/union the results at the application level if we needed to > search over both active and archived data? > > On Thursday, April 26, 2012 10:57:50 AM UTC-7, Berkay Mollamustafaoglu > wrote: > > > No worries. Yes I'd recommend 2 separate indices not types. This would > > allow you to optimize them differently. There are number of params that you > > can only set in index level. > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Thu, Apr 26, 2012 at 1:19 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > > > Sorry for my late reply, for some reason I was not notified when you > > > posted. Documents can move from active to archived at any time so we will > > > know as the document is indexed or updated where it belongs. In our > > > database we split the records into two separate tables for performance > > > reasons. > > > > > > So your suggestion is to create a completely separate index or type? > > > Sorry, I'm fairly new to the Elasticsearch terminology... > > > > > > Thanks, > > > > > > Sky > > > > > > On Tuesday, April 24, 2012 8:40:37 AM UTC-7, Berkay Mollamustafaoglu > > > wrote: > > > > > > > I'd consider using separate indices and aliases. Keeping the active > > > > index smaller would help with the performance. Will you know before you > > > > index which docs are archived data and which are active? > > > > > > > > Regards, > > > > Berkay Mollamustafaoglu > > > > mberkay on yahoo, google and skype > > > > > > > > On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki \< > > > > [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)\> wrote: > > > > > > > > > Hi all, > > > > > > > > > > We've been testing elasticsearch with our application and are really > > > > > enjoying impressive performance and rock-solid stability. I have a design > > > > > question about what would be the most efficient way to index (1) large set > > > > > of highly active date and (2) an even larger set of archived > > > > > data. Basically we have tens of millions of documents but about 80% of them > > > > > are in archive state and 10-20% are read/updated 95% of the time. > > > > > > > > > > My question is this: would it be more efficient to store the archived > > > > > documents into a separate type like "/index/mydata\_arch" or to just use a > > > > > filtered query to cache the results and flag the archived documents as we > > > > > index them? > > > > > > > > > > We are working on setting up benchmarks to test this ourselves in a > > > > > real-word environment but I wanted to ask the experts here too and see if > > > > > you had any input. > > > > > > > > > > Thanks so much for your help! > > > > > > > > > > Sky ---
**Author:** ![Sky\_Stebnicki1](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@Sky\_Stebnicki1](https://discuss.elastic.co/u/Sky_Stebnicki1)\ **Post date:** [April 27, 2012, 5:59pm UTC](https://discuss.elastic.co/t/large-index-design-question/7454/7 "2012-04-27T17:59:28Z")
That is really awesome! Thanks so much for your help. On Thursday, April 26, 2012 7:28:46 PM UTC-7, Berkay Mollamustafaoglu wrote: > You don't have to merge the results yourself. ES allows you create an > alias that references to multiple indices and then query the alias like a > normal index. It'll do the work for you. They are quite powerful and > useful. Details here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html) > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Thu, Apr 26, 2012 at 7:46 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > What did you mean by "aliases"? If we use two separate indexes would I > > have to merge/union the results at the application level if we needed to > > search over both active and archived data? > > > > On Thursday, April 26, 2012 10:57:50 AM UTC-7, Berkay Mollamustafaoglu > > wrote: > > > > > No worries. Yes I'd recommend 2 separate indices not types. This would > > > allow you to optimize them differently. There are number of params that you > > > can only set in index level. > > > > > > Regards, > > > Berkay Mollamustafaoglu > > > mberkay on yahoo, google and skype > > > > > > On Thu, Apr 26, 2012 at 1:19 PM, Sky Stebnicki [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)wrote: > > > > > > > Sorry for my late reply, for some reason I was not notified when you > > > > posted. Documents can move from active to archived at any time so we will > > > > know as the document is indexed or updated where it belongs. In our > > > > database we split the records into two separate tables for performance > > > > reasons. > > > > > > > > So your suggestion is to create a completely separate index or type? > > > > Sorry, I'm fairly new to the Elasticsearch terminology... > > > > > > > > Thanks, > > > > > > > > Sky > > > > > > > > On Tuesday, April 24, 2012 8:40:37 AM UTC-7, Berkay Mollamustafaoglu > > > > wrote: > > > > > > > > > I'd consider using separate indices and aliases. Keeping the active > > > > > index smaller would help with the performance. Will you know before you > > > > > index which docs are archived data and which are active? > > > > > > > > > > Regards, > > > > > Berkay Mollamustafaoglu > > > > > mberkay on yahoo, google and skype > > > > > > > > > > On Tue, Apr 24, 2012 at 10:21 AM, Sky Stebnicki \< > > > > > [sky.stebnicki@gmail.com](mailto:sky.stebnicki@gmail.com)\> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > We've been testing elasticsearch with our application and are really > > > > > > enjoying impressive performance and rock-solid stability. I have a design > > > > > > question about what would be the most efficient way to index (1) large set > > > > > > of highly active date and (2) an even larger set of archived > > > > > > data. Basically we have tens of millions of documents but about 80% of them > > > > > > are in archive state and 10-20% are read/updated 95% of the time. > > > > > > > > > > > > My question is this: would it be more efficient to store the archived > > > > > > documents into a separate type like "/index/mydata\_arch" or to just use a > > > > > > filtered query to cache the results and flag the archived documents as we > > > > > > index them? > > > > > > > > > > > > We are working on setting up benchmarks to test this ourselves in a > > > > > > real-word environment but I wanted to ask the experts here too and see if > > > > > > you had any input. > > > > > > > > > > > > Thanks so much for your help! > > > > > > > > > > > > Sky ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:30am UTC](https://discuss.elastic.co/t/large-index-design-question/7454/8 "2017-07-06T03:30:50Z")