# 24/7 Index Service? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 17, 2011, 9:31am UTC](https://discuss.elastic.co/t/24-7-index-service/3787 "2011-01-17T09:31:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [January 17, 2011, 9:31am UTC](https://discuss.elastic.co/t/24-7-index-service/3787/1 "2011-01-17T09:31:20Z")
Hi, What is the best clustering / indexing strategy for making a large batch update / re-indexing on a nightly basis so that users' search performance is not affected. The update would be approximately 4 millioin documents. thanks. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 17, 2011, 5:10pm UTC](https://discuss.elastic.co/t/24-7-index-service/3787/2 "2011-01-17T17:10:35Z")
If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > Hi, > > What is the best clustering / indexing strategy for making a large > batch update / re-indexing on a nightly basis so that users' search > performance is not affected. The update would be approximately 4 > millioin documents. > > thanks. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 17, 2011, 6:14pm UTC](https://discuss.elastic.co/t/24-7-index-service/3787/3 "2011-01-17T18:14:01Z")
Could one tune this scenario? like decreasing realtime? E.g. with solr I can index into indexA and let it replicate from that into indexB. The queries then goes against indexB (of course realtime is then really bad ...) Regards, Peter. On 17 Jan., 18:10, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. > > On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > > > Hi, > > > What is the best clustering / indexing strategy for making a large > > batch update / re-indexing on a nightly basis so that users' search > > performance is not affected. The update would be approximately 4 > > millioin documents. > > > thanks. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 17, 2011, 6:52pm UTC](https://discuss.elastic.co/t/24-7-index-service/3787/4 "2011-01-17T18:52:14Z")
There has been some talk on the issues to allow for shard replicas that only sync on index flushes (i.e. don't do the index operation, instead sync on the index actual files, which are updated on a flush in ES). Before anyone does any optimizations, I think a good place to start is to see if you really need them at all. ES is used in some big systems, and it holds its own pretty well. On Monday, January 17, 2011 at 8:14 PM, Karussell wrote: > Could one tune this scenario? like decreasing realtime? > > E.g. with solr I can index into indexA and let it replicate from that > into indexB. The queries then goes against indexB (of course realtime > is then really bad ...) > > Regards, > Peter. > > On 17 Jan., 18:10, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. > > > > On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > > > > > Hi, > > > > > What is the best clustering / indexing strategy for making a large > > > batch update / re-indexing on a nightly basis so that users' search > > > performance is not affected. The update would be approximately 4 > > > millioin documents. > > > > > thanks. ---
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [January 18, 2011, 10:27am UTC](https://discuss.elastic.co/t/24-7-index-service/3787/5 "2011-01-18T10:27:27Z")
Hi Shay, Thanks for your quick answer, here are a few questions I have around this: 1. If create the nightly index as a new index name how do I then make it searchable under the old index, do I just create an alias at the end of the indexing process to alias the new index name to the old one? 2. At some point wouldn't I also have to delete or decomission the old index, how much load would that have on the server, should I just do a searchAll query and delete that way? 3. Is it possible to create a server just for indexing and at some point tell it to merge into the rest of the cluster. Can you give me some guidance on how I would do that the clustering is definitely just an out-of-the-box thing for me at the moment. David. On Jan 17, 5:10 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. > > On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > > > Hi, > > > What is the best clustering / indexing strategy for making a large > > batch update / re-indexing on a nightly basis so that users' search > > performance is not affected. The update would be approximately 4 > > millioin documents. > > > thanks. ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [January 18, 2011, 4:40pm UTC](https://discuss.elastic.co/t/24-7-index-service/3787/6 "2011-01-18T16:40:17Z")
We don't rebuild our indexes nightly (not sure why you need to do that), but do have a mechanism in place for occasional index rebuilds. ES supports the notion of a physical index and an alias. We never reference the physical index in our queries, just the alias. We search against index01 ([http://localhost:9200/index01/\_search](http://localhost:9200/index01/_search)...) Which actually points at: indices: { -index01\_20110105224045: { -aliases: [ "index01" ] The physical index name is the index name + date/time stamp. When we're rebuilding content we have: indices: { -index01\_20110115224045: { -aliases: [ "rebuild\_index01" ] Then when the rebuilt index is complete, we switch the aliases. Then we drop the old physical index. Its more efficient to drop the index, than to delete by query. David On Jan 18, 3:27 am, dalesrob [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote: > Hi Shay, > > Thanks for your quick answer, here are a few questions I have around > this: > > 1. If create the nightly index as a new index name how do I then make > it searchable under the old index, do I just create an alias at the > end of the indexing process to alias the new index name to the old > one? > > 2. At some point wouldn't I also have to delete or decomission the > old index, how much load would that have on the server, should I just > do a searchAll query and delete that way? > > 3. Is it possible to create a server just for indexing and at some > point tell it to merge into the rest of the cluster. Can you give me > some guidance on how I would do that the clustering is definitely just > an out-of-the-box thing for me at the moment. > > David. > > On Jan 17, 5:10 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. > > > On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > > > > > Hi, > > > > What is the best clustering / indexing strategy for making a large > > > batch update / re-indexing on a nightly basis so that users' search > > > performance is not affected. The update would be approximately 4 > > > millioin documents. > > > > thanks. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 18, 2011, 6:30pm UTC](https://discuss.elastic.co/t/24-7-index-service/3787/7 "2011-01-18T18:30:04Z")
dbenson nailed most of the answers. Again, I really recommend testing if you get a high load in this case to really require a complete reindex into a fresh index. On Tuesday, January 18, 2011 at 6:40 PM, dbenson wrote: > We don't rebuild our indexes nightly (not sure why you need to do > that), but do have a mechanism in place for occasional index > rebuilds. > > ES supports the notion of a physical index and an alias. We never > reference the physical index in our queries, just the alias. > > We search against index01 ([http://localhost:9200/index01/\_search](http://localhost:9200/index01/_search)...) > Which actually points at: > indices: { > -index01\_20110105224045: { > -aliases: [ > "index01" > ] > > The physical index name is the index name + date/time stamp. When > we're rebuilding content we have: > indices: { > -index01\_20110115224045: { > -aliases: [ > "rebuild\_index01" > ] > > Then when the rebuilt index is complete, we switch the aliases. Then > we drop the old physical index. Its more efficient to drop the index, > than to delete by query. > > David > > On Jan 18, 3:27 am, dalesrob [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote: > > > Hi Shay, > > > > Thanks for your quick answer, here are a few questions I have around > > this: > > > > 1. If create the nightly index as a new index name how do I then make > > it searchable under the old index, do I just create an alias at the > > end of the indexing process to alias the new index name to the old > > one? > > > > 2. At some point wouldn't I also have to delete or decomission the > > old index, how much load would that have on the server, should I just > > do a searchAll query and delete that way? > > > > 3. Is it possible to create a server just for indexing and at some > > point tell it to merge into the rest of the cluster. Can you give me > > some guidance on how I would do that the clustering is definitely just > > an out-of-the-box thing for me at the moment. > > > > David. > > > > On Jan 17, 5:10 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > If you end up updating most of the docs, you can index the data into a different index. If not, then it will create more load on the system, just make sure you have enough servers to handle the load for both indexing and search. > > > > > On Monday, January 17, 2011 at 11:31 AM, dalesrob wrote: > > > > > > > Hi, > > > > > > What is the best clustering / indexing strategy for making a large > > > > batch update / re-indexing on a nightly basis so that users' search > > > > performance is not affected. The update would be approximately 4 > > > > millioin documents. > > > > > > thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:13am UTC](https://discuss.elastic.co/t/24-7-index-service/3787/8 "2017-07-06T04:13:35Z")