In memory node with guaranteed local data

Elastic Stack Elasticsearch
1

TL;DR: Is there a way to reliably guarantee that a local node has all the
data of an index locally in memory?

We have a very time critical application (search times > 5-10 ms are a real
problem for us). To keep search as fast as possible we figured the best way
would be to use a node in the same process to eliminate network latency and
also being as self contained as possible.
That's working out nicely, so far, except that we have to manually change
the number_of_replicas every time we add or remove a node to the cluster.
We actually don't want to mess with the replica settings at all.
We just want our local nodes to join the cluster, grab *all *the data for
the index so we never ever have to ask additional nodes again.

Bonus question: are there any performance does and don'ts hints / links /
blogposts / best practices you could recommend?

thank you!

--

2

If I understood your question correctly, you can set index.auto_expand_replicas to all, in which case it will make sure to place replicas automatically on all the data nodes based on the current data nodes in the cluster.

On Oct 11, 2012, at 7:05 AM, Pascal Friederich paukul@gmail.com wrote:

TL;DR: Is there a way to reliably guarantee that a local node has all the data of an index locally in memory?

We have a very time critical application (search times > 5-10 ms are a real problem for us). To keep search as fast as possible we figured the best way would be to use a node in the same process to eliminate network latency and also being as self contained as possible.
That's working out nicely, so far, except that we have to manually change the number_of_replicas every time we add or remove a node to the cluster.
We actually don't want to mess with the replica settings at all.
We just want our local nodes to join the cluster, grab all the data for the index so we never ever have to ask additional nodes again.

Bonus question: are there any performance does and don'ts hints / links / blogposts / best practices you could recommend?

thank you!

--

--

3

perfect, thanks. I actually saw that option but didn't make the
connection...

On Thursday, October 11, 2012 5:06:36 PM UTC+2, kimchy wrote:

If I understood your question correctly, you can
set index.auto_expand_replicas to all, in which case it will make sure to
place replicas automatically on all the data nodes based on the current
data nodes in the cluster.

On Oct 11, 2012, at 7:05 AM, Pascal Friederich <pau...@gmail.com<javascript:>>
wrote:

TL;DR: Is there a way to reliably guarantee that a local node has all the
data of an index locally in memory?

We have a very time critical application (search times > 5-10 ms are a
real problem for us). To keep search as fast as possible we figured the
best way would be to use a node in the same process to eliminate network
latency and also being as self contained as possible.
That's working out nicely, so far, except that we have to manually change
the number_of_replicas every time we add or remove a node to the cluster.
We actually don't want to mess with the replica settings at all.
We just want our local nodes to join the cluster, grab *all *the data for
the index so we never ever have to ask additional nodes again.

Bonus question: are there any performance does and don'ts hints / links /
blogposts / best practices you could recommend?

thank you!

--

--