How to determine optimum RAM for an elasticsearch node


(subin) #1

Hello,

I've been pondering over this for some time. Obviously RAM will depend on
the total amount of indexes to span over. But what else are the other
factors to consider? Assuming that at any time, ES has to search through 30
days of indexes, amounting to about 250G, how could I take up calculations
from here? let's say a single node with 1 shard/0 replica?

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #2

It depends! :slight_smile:

Seriously, facets are memory intensive for example. Sorting as well.
So the answer is always: test it with your queries.

(I know: it does not really help :frowning: )

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 oct. 2013 à 08:30, subin ksubins321@gmail.com a écrit :

Hello,

I've been pondering over this for some time. Obviously RAM will depend on the total amount of indexes to span over. But what else are the other factors to consider? Assuming that at any time, ES has to search through 30 days of indexes, amounting to about 250G, how could I take up calculations from here? let's say a single node with 1 shard/0 replica?

Thanks,

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #3

David, this is the template that I use -
https://gist.github.com/osssubb/6774727. There is no faceting to transform
my data. I use Kibana to search the logs and sorting, is done on @timestamp
field. Now, could there by an clues or pointers, still? Because I see
optimal usage when no searches are being done but the usage of Heap memory
shoots up when searches are done and the events start getting queued up on
REDIS. In normal cases, when REDIS's queues are cleared off the memory
usage is like 4-5Mb. When searches are done on ES(from Kibana) it grows to
30Mb+.


Subin
*
*

On Tue, Oct 22, 2013 at 12:09 PM, David Pilato david@pilato.fr wrote:

It depends! :slight_smile:

Seriously, facets are memory intensive for example. Sorting as well.
So the answer is always: test it with your queries.

(I know: it does not really help :frowning: )

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 oct. 2013 à 08:30, subin ksubins321@gmail.com a écrit :

Hello,

I've been pondering over this for some time. Obviously RAM will depend on
the total amount of indexes to span over. But what else are the other
factors to consider? Assuming that at any time, ES has to search through 30
days of indexes, amounting to about 250G, how could I take up calculations
from here? let's say a single node with 1 shard/0 replica?

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7o2ZjRUYziE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #4

from bigdesk I'm able to see an average of 1000 for indexing requests/s...

Thanks!


Subin
*
*

On Thu, Oct 24, 2013 at 8:46 PM, subin ks ksubins321@gmail.com wrote:

David, this is the template that I use -
https://gist.github.com/osssubb/6774727. There is no faceting to
transform my data. I use Kibana to search the logs and sorting, is done on
@timestamp field. Now, could there by an clues or pointers, still? Because
I see optimal usage when no searches are being done but the usage of Heap
memory shoots up when searches are done and the events start getting queued
up on REDIS. In normal cases, when REDIS's queues are cleared off the
memory usage is like 4-5Mb. When searches are done on ES(from Kibana) it
grows to 30Mb+.


Subin
*
*

On Tue, Oct 22, 2013 at 12:09 PM, David Pilato david@pilato.fr wrote:

It depends! :slight_smile:

Seriously, facets are memory intensive for example. Sorting as well.
So the answer is always: test it with your queries.

(I know: it does not really help :frowning: )

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 oct. 2013 à 08:30, subin ksubins321@gmail.com a écrit :

Hello,

I've been pondering over this for some time. Obviously RAM will depend on
the total amount of indexes to span over. But what else are the other
factors to consider? Assuming that at any time, ES has to search through 30
days of indexes, amounting to about 250G, how could I take up calculations
from here? let's say a single node with 1 shard/0 replica?

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7o2ZjRUYziE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #5

So you use Kibana? Then probably behind the scene you use facets (histogram panel, terms panel…)
Facets use a lot of memory.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 24 oct. 2013 à 17:16, subin ks ksubins321@gmail.com a écrit :

David, this is the template that I use - https://gist.github.com/osssubb/6774727. There is no faceting to transform my data. I use Kibana to search the logs and sorting, is done on @timestamp field. Now, could there by an clues or pointers, still? Because I see optimal usage when no searches are being done but the usage of Heap memory shoots up when searches are done and the events start getting queued up on REDIS. In normal cases, when REDIS's queues are cleared off the memory usage is like 4-5Mb. When searches are done on ES(from Kibana) it grows to 30Mb+.


Subin

On Tue, Oct 22, 2013 at 12:09 PM, David Pilato david@pilato.fr wrote:
It depends! :slight_smile:

Seriously, facets are memory intensive for example. Sorting as well.
So the answer is always: test it with your queries.

(I know: it does not really help :frowning: )

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 oct. 2013 à 08:30, subin ksubins321@gmail.com a écrit :

Hello,

I've been pondering over this for some time. Obviously RAM will depend on the total amount of indexes to span over. But what else are the other factors to consider? Assuming that at any time, ES has to search through 30 days of indexes, amounting to about 250G, how could I take up calculations from here? let's say a single node with 1 shard/0 replica?

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/7o2ZjRUYziE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6