High availability for last N feeds


(vineeth mohan) #1

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to pick
it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in the
main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while searching
(Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(vineeth mohan) #2

Going through the documents i found
cache.memory.small_cache_size

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan vineethmohan@algotree.comwrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to pick
it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in
the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(Shay Banon) #3

There isn't a way to cache the "last" docs, but if you access them a lot,
the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example, adding a
flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan vineethmohan@algotree.comwrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan vineethmohan@algotree.comwrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to pick
it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in
the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(vineeth mohan) #4

I got that setting from -
http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all
feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:

There isn't a way to cache the "last" docs, but if you access them a lot,
the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example, adding a
flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan vineethmohan@algotree.comwrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan <vineethmohan@algotree.com

wrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to
pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in
the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(Shay Banon) #5

That setting will no longer be relevant in 0.18, it did not do much. No
control over what gets cached. File system cache is managed by the OS.

On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

I got that setting from -
http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all
feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:

There isn't a way to cache the "last" docs, but if you access them a lot,
the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example, adding
a flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan <vineethmohan@algotree.com

wrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to
pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in
the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(vineeth mohan) #6

So is there provision to write some plugin which decide what gets cached.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:

That setting will no longer be relevant in 0.18, it did not do much. No
control over what gets cached. File system cache is managed by the OS.

On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

I got that setting from -
http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all
feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:

There isn't a way to cache the "last" docs, but if you access them a lot,
the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example, adding
a flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to
pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be
in the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be
searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(Shay Banon) #7

No, but I suspect you are trying to optimize thins even before you know
there is a problem, no?

On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

So is there provision to write some plugin which decide what gets cached.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:

That setting will no longer be relevant in 0.18, it did not do much. No
control over what gets cached. File system cache is managed by the OS.

On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan <vineethmohan@algotree.com

wrote:

I got that setting from -
http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all
feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:

There isn't a way to cache the "last" docs, but if you access them a
lot, the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example,
adding a flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to
pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be
in the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it
be searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(vineeth mohan) #8

yes, we haven't yet hit the production.
Just looking around to see how to make things more efficient before hitting
production.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 10:13 AM, Shay Banon kimchy@gmail.com wrote:

No, but I suspect you are trying to optimize thins even before you know
there is a problem, no?

On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

So is there provision to write some plugin which decide what gets cached.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:

That setting will no longer be relevant in 0.18, it did not do much. No
control over what gets cached. File system cache is managed by the OS.

On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

I got that setting from -
http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all
feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:

There isn't a way to cache the "last" docs, but if you access them a
lot, the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example,
adding a flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan <
vineethmohan@algotree.com> wrote:

Hi ,

ES will serve as the integration point for various application in our
system.
One of the component uploads feeds to ES and ask another component to
pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be
in the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it
be searchable.
One of the way would be to filter out un processed documents while
searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(David Pilato) #9

I think that Shay means : "Don't try to solve problems you don't have !"
The risk is that you create problems yourself !

But, before going in production, do some stress tests to see if there are problems and only in that case, try to solve them.

Cheers,
David :wink:

Le 26 oct. 2011 à 11:00, Vineeth Mohan vineethmohan@algotree.com a écrit :

yes, we haven't yet hit the production.
Just looking around to see how to make things more efficient before hitting production.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 10:13 AM, Shay Banon kimchy@gmail.com wrote:
No, but I suspect you are trying to optimize thins even before you know there is a problem, no?

On Wed, Oct 26, 2011 at 6:26 AM, Vineeth Mohan vineethmohan@algotree.com wrote:
So is there provision to write some plugin which decide what gets cached.

Thanks
Vineeth

On Wed, Oct 26, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:
That setting will no longer be relevant in 0.18, it did not do much. No control over what gets cached. File system cache is managed by the OS.

On Wed, Oct 26, 2011 at 5:51 AM, Vineeth Mohan vineethmohan@algotree.com wrote:
I got that setting from - http://www.elasticsearch.org/guide/reference/index-modules/store.html

So is there anyway i can control the algorithm used to decide which all feeds needs to be cached ?

Thanks
Vineeth

On Wed, Oct 26, 2011 at 8:30 AM, Shay Banon kimchy@gmail.com wrote:
There isn't a way to cache the "last" docs, but if you access them a lot, the file system cache should be there to speed things up.

Regarding filtering, you will have to do it yourself, for example, adding a flag in the document, and only searching when the flag is set.

On Tue, Oct 25, 2011 at 4:58 PM, Vineeth Mohan vineethmohan@algotree.com wrote:
Going through the documents i found
cache.memory.small_cache_size

Which setting is that?

I hope this will serve my purpose.

Is there any way out for the latter part of my question ?
Basically i want to control when a document will turn searchable

Thanks
Vineeth

On Tue, Oct 25, 2011 at 8:20 PM, Vineeth Mohan vineethmohan@algotree.com wrote:
Hi ,

ES will serve as the integration point for various application in our system.
One of the component uploads feeds to ES and ask another component to pick it up (It passes the docID).
Is it possible to make last N feeds given to a particular index to be in the main memory.
So that this process is executed in a faster manner.

Also only after all components adds details to a document, should it be searchable.
One of the way would be to filter out un processed documents while searching (Which i am trying not to do)

Is there any other better solution to this ?

Thanks
Vineeth


(system) #10