# High availability for last N feeds **URL:** \ **Category:** Elasticsearch\ **Created:** [October 25, 2011, 2:50pm UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675 "2011-10-25T14:50:50Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 25, 2011, 2:50pm UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/1 "2011-10-25T14:50:50Z")
Hi , ES will serve as the integration point for various application in our system. One of the component uploads feeds to ES and ask another component to pick it up (It passes the docID). Is it possible to make last N feeds given to a particular index to be in the main memory. So that this process is executed in a faster manner. Also only after all components adds details to a document, should it be searchable. One of the way would be to filter out un processed documents while searching (Which i am trying not to do) Is there any other better solution to this ? Thanks Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 25, 2011, 2:58pm UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/2 "2011-10-25T14:58:35Z")
Going through the documents i found cache.memory.small\_cache\_size I hope this will serve my purpose. Is there any way out for the latter part of my question ? Basically i want to control when a document will turn searchable Thanks Vineeth On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Hi , > > ES will serve as the integration point for various application in our > system. > One of the component uploads feeds to ES and ask another component to pick > it up (It passes the docID). > Is it possible to make last N feeds given to a particular index to be in > the main memory. > So that this process is executed in a faster manner. > > Also only after all components adds details to a document, should it be > searchable. > One of the way would be to filter out un processed documents while > searching (Which i am trying not to do) > > Is there any other better solution to this ? > > Thanks > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 3:00am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/3 "2011-10-26T03:00:58Z")
There isn't a way to cache the "last" docs, but if you access them a lot, the file system cache should be there to speed things up. Regarding filtering, you will have to do it yourself, for example, adding a flag in the document, and only searching when the flag is set. On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > Going through the documents i found > cache.memory.small\_cache\_size Which setting is that? > I hope this will serve my purpose. > > Is there any way out for the latter part of my question ? > Basically i want to control when a document will turn searchable > > Thanks > Vineeth > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Hi , > > > > ES will serve as the integration point for various application in our > > system. > > One of the component uploads feeds to ES and ask another component to pick > > it up (It passes the docID). > > Is it possible to make last N feeds given to a particular index to be in > > the main memory. > > So that this process is executed in a faster manner. > > > > Also only after all components adds details to a document, should it be > > searchable. > > One of the way would be to filter out un processed documents while > > searching (Which i am trying not to do) > > > > Is there any other better solution to this ? > > > > Thanks > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 26, 2011, 3:51am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/4 "2011-10-26T03:51:48Z")
I got that setting from - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. So is there anyway i can control the algorithm used to decide which all feeds needs to be cached ? Thanks Vineeth On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > There isn't a way to cache the "last" docs, but if you access them a lot, > the file system cache should be there to speed things up. > > Regarding filtering, you will have to do it yourself, for example, adding a > flag in the document, and only searching when the flag is set. > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > Going through the documents i found > > cache.memory.small\_cache\_size > > Which setting is that? > > > I hope this will serve my purpose. > > > > Is there any way out for the latter part of my question ? > > Basically i want to control when a document will turn searchable > > > > Thanks > > Vineeth > > > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > > > > > wrote: > > > > > Hi , > > > > > > ES will serve as the integration point for various application in our > > > system. > > > One of the component uploads feeds to ES and ask another component to > > > pick it up (It passes the docID). > > > Is it possible to make last N feeds given to a particular index to be in > > > the main memory. > > > So that this process is executed in a faster manner. > > > > > > Also only after all components adds details to a document, should it be > > > searchable. > > > One of the way would be to filter out un processed documents while > > > searching (Which i am trying not to do) > > > > > > Is there any other better solution to this ? > > > > > > Thanks > > > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 4:23am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/5 "2011-10-26T04:23:11Z")
That setting will no longer be relevant in 0.18, it did not do much. No control over what gets cached. File system cache is managed by the OS. On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > I got that setting from - > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > So is there anyway i can control the algorithm used to decide which all > feeds needs to be cached ? > > Thanks > Vineeth > > On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > There isn't a way to cache the "last" docs, but if you access them a lot, > > the file system cache should be there to speed things up. > > > > Regarding filtering, you will have to do it yourself, for example, adding > > a flag in the document, and only searching when the flag is set. > > > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > > > > > wrote: > > > > > Going through the documents i found > > > cache.memory.small\_cache\_size > > > > Which setting is that? > > > > > I hope this will serve my purpose. > > > > > > Is there any way out for the latter part of my question ? > > > Basically i want to control when a document will turn searchable > > > > > > Thanks > > > Vineeth > > > > > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan \< > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > Hi , > > > > > > > > ES will serve as the integration point for various application in our > > > > system. > > > > One of the component uploads feeds to ES and ask another component to > > > > pick it up (It passes the docID). > > > > Is it possible to make last N feeds given to a particular index to be in > > > > the main memory. > > > > So that this process is executed in a faster manner. > > > > > > > > Also only after all components adds details to a document, should it be > > > > searchable. > > > > One of the way would be to filter out un processed documents while > > > > searching (Which i am trying not to do) > > > > > > > > Is there any other better solution to this ? > > > > > > > > Thanks > > > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 26, 2011, 4:26am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/6 "2011-10-26T04:26:32Z")
So is there provision to write some plugin which decide what gets cached. Thanks Vineeth On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > That setting will no longer be relevant in 0.18, it did not do much. No > control over what gets cached. File system cache is managed by the OS. > > On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > I got that setting from - > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > So is there anyway i can control the algorithm used to decide which all > > feeds needs to be cached ? > > > > Thanks > > Vineeth > > > > On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > There isn't a way to cache the "last" docs, but if you access them a lot, > > > the file system cache should be there to speed things up. > > > > > > Regarding filtering, you will have to do it yourself, for example, adding > > > a flag in the document, and only searching when the flag is set. > > > > > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan \< > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > Going through the documents i found > > > > cache.memory.small\_cache\_size > > > > > > Which setting is that? > > > > > > > I hope this will serve my purpose. > > > > > > > > Is there any way out for the latter part of my question ? > > > > Basically i want to control when a document will turn searchable > > > > > > > > Thanks > > > > Vineeth > > > > > > > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan \< > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > > > Hi , > > > > > > > > > > ES will serve as the integration point for various application in our > > > > > system. > > > > > One of the component uploads feeds to ES and ask another component to > > > > > pick it up (It passes the docID). > > > > > Is it possible to make last N feeds given to a particular index to be > > > > > in the main memory. > > > > > So that this process is executed in a faster manner. > > > > > > > > > > Also only after all components adds details to a document, should it be > > > > > searchable. > > > > > One of the way would be to filter out un processed documents while > > > > > searching (Which i am trying not to do) > > > > > > > > > > Is there any other better solution to this ? > > > > > > > > > > Thanks > > > > > Vineeth ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 4:43am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/7 "2011-10-26T04:43:02Z")
No, but I suspect you are trying to optimize thins even before you know there is a problem, no? On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > So is there provision to write some plugin which decide what gets cached. > > Thanks > Vineeth > > On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > That setting will no longer be relevant in 0.18, it did not do much. No > > control over what gets cached. File system cache is managed by the OS. > > > > On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) > > > > > wrote: > > > > > I got that setting from - > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > So is there anyway i can control the algorithm used to decide which all > > > feeds needs to be cached ? > > > > > > Thanks > > > Vineeth > > > > > > On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > There isn't a way to cache the "last" docs, but if you access them a > > > > lot, the file system cache should be there to speed things up. > > > > > > > > Regarding filtering, you will have to do it yourself, for example, > > > > adding a flag in the document, and only searching when the flag is set. > > > > > > > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan \< > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > > > Going through the documents i found > > > > > cache.memory.small\_cache\_size > > > > > > > > Which setting is that? > > > > > > > > > I hope this will serve my purpose. > > > > > > > > > > Is there any way out for the latter part of my question ? > > > > > Basically i want to control when a document will turn searchable > > > > > > > > > > Thanks > > > > > Vineeth > > > > > > > > > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan \< > > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > > > > > Hi , > > > > > > > > > > > > ES will serve as the integration point for various application in our > > > > > > system. > > > > > > One of the component uploads feeds to ES and ask another component to > > > > > > pick it up (It passes the docID). > > > > > > Is it possible to make last N feeds given to a particular index to be > > > > > > in the main memory. > > > > > > So that this process is executed in a faster manner. > > > > > > > > > > > > Also only after all components adds details to a document, should it > > > > > > be searchable. > > > > > > One of the way would be to filter out un processed documents while > > > > > > searching (Which i am trying not to do) > > > > > > > > > > > > Is there any other better solution to this ? > > > > > > > > > > > > Thanks > > > > > > Vineeth ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [October 26, 2011, 9:00am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/8 "2011-10-26T09:00:25Z")
yes, we haven't yet hit the production. Just looking around to see how to make things more efficient before hitting production. Thanks Vineeth On Wed, Oct 26, 2011 at 10:13 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > No, but I suspect you are trying to optimize thins even before you know > there is a problem, no? > > On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote: > > > So is there provision to write some plugin which decide what gets cached. > > > > Thanks > > Vineeth > > > > On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > That setting will no longer be relevant in 0.18, it did not do much. No > > > control over what gets cached. File system cache is managed by the OS. > > > > > > On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan \< > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > I got that setting from - > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > > > So is there anyway i can control the algorithm used to decide which all > > > > feeds needs to be cached ? > > > > > > > > Thanks > > > > Vineeth > > > > > > > > On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > There isn't a way to cache the "last" docs, but if you access them a > > > > > lot, the file system cache should be there to speed things up. > > > > > > > > > > Regarding filtering, you will have to do it yourself, for example, > > > > > adding a flag in the document, and only searching when the flag is set. > > > > > > > > > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan \< > > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > > > > > Going through the documents i found > > > > > > cache.memory.small\_cache\_size > > > > > > > > > > Which setting is that? > > > > > > > > > > > I hope this will serve my purpose. > > > > > > > > > > > > Is there any way out for the latter part of my question ? > > > > > > Basically i want to control when a document will turn searchable > > > > > > > > > > > > Thanks > > > > > > Vineeth > > > > > > > > > > > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan \< > > > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote: > > > > > > > > > > > > > Hi , > > > > > > > > > > > > > > ES will serve as the integration point for various application in our > > > > > > > system. > > > > > > > One of the component uploads feeds to ES and ask another component to > > > > > > > pick it up (It passes the docID). > > > > > > > Is it possible to make last N feeds given to a particular index to be > > > > > > > in the main memory. > > > > > > > So that this process is executed in a faster manner. > > > > > > > > > > > > > > Also only after all components adds details to a document, should it > > > > > > > be searchable. > > > > > > > One of the way would be to filter out un processed documents while > > > > > > > searching (Which i am trying not to do) > > > > > > > > > > > > > > Is there any other better solution to this ? > > > > > > > > > > > > > > Thanks > > > > > > > Vineeth ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 26, 2011, 9:09am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/9 "2011-10-26T09:09:40Z")
I think that Shay means : "Don't try to solve problems you don't have !" The risk is that you create problems yourself ! But, before going in production, do some stress tests to see if there are problems and only in that case, try to solve them. Cheers, David 😉 Le 26 oct. 2011 à 11:00, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) a écrit : > yes, we haven't yet hit the production. > Just looking around to see how to make things more efficient before hitting production. > > Thanks > Vineeth > > On Wed, Oct 26, 2011 at 10:13 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > No, but I suspect you are trying to optimize thins even before you know there is a problem, no? > > On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > So is there provision to write some plugin which decide what gets cached. > > Thanks > Vineeth > > On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > That setting will no longer be relevant in 0.18, it did not do much. No control over what gets cached. File system cache is managed by the OS. > > On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > I got that setting from - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > So is there anyway i can control the algorithm used to decide which all feeds needs to be cached ? > > Thanks > Vineeth > > On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > There isn't a way to cache the "last" docs, but if you access them a lot, the file system cache should be there to speed things up. > > Regarding filtering, you will have to do it yourself, for example, adding a flag in the document, and only searching when the flag is set. > > On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > Going through the documents i found > cache.memory.small\_cache\_size > > Which setting is that? > > I hope this will serve my purpose. > > Is there any way out for the latter part of my question ? > Basically i want to control when a document will turn searchable > > Thanks > Vineeth > > On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) wrote: > Hi , > > ES will serve as the integration point for various application in our system. > One of the component uploads feeds to ES and ask another component to pick it up (It passes the docID). > Is it possible to make last N feeds given to a particular index to be in the main memory. > So that this process is executed in a faster manner. > > Also only after all components adds details to a document, should it be searchable. > One of the way would be to filter out un processed documents while searching (Which i am trying not to do) > > Is there any other better solution to this ? > > Thanks > Vineeth ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/high-availability-for-last-n-feeds/5675/10 "2017-07-06T03:50:55Z")