ES Memory consuption

Hi,

I'm having troubles managing the memory on my ES server. I have 3 nodes
with heap size 4GB each. However, when i start indexing my memory start
growing proportionally to the size of my index. In detail, if I run "sar
-r" i see that memory used doesn't go more than 12GB which is ok, but
my kbcached continous growing. How can I limit that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look unnecessary or
simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Did you run into troubles? Are there messages in the logs? What is your
challenge? Why do you want to limit what kind of cache?

It is not very clear how to help - please, do not "press buttons" out of
despair, it will not help without understanding what is going on.

Jörg

Am 20.03.13 19:00, schrieb Delmar:

Hi,

I'm having troubles managing the memory on my ES server. I have 3
nodes with heap size 4GB each. However, when i start indexing my
memory start growing proportionally to the size of my index. In
detail, if I run "sar -r" i see that memory used doesn't go more than
12GB which is ok, but my kbcached continous growing. How can I limit that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look
unnecessary or simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I'm trying to understand, that's why I tried different aproaches.
My problem is that i want to limit the size of the cache because there
might be running other applications on the same server that might need
resources also. I don't want ES to own or try to own all the RAM resources
of my server. Nevertheless, is it possible to limit it? If so, how can i do
it?
Thank's

Quarta-feira, 20 de Março de 2013 18:14:00 UTC, Jörg Prante escreveu:

Did you run into troubles? Are there messages in the logs? What is your
challenge? Why do you want to limit what kind of cache?

It is not very clear how to help - please, do not "press buttons" out of
despair, it will not help without understanding what is going on.

Jörg

Am 20.03.13 19:00, schrieb Delmar:

Hi,

I'm having troubles managing the memory on my ES server. I have 3
nodes with heap size 4GB each. However, when i start indexing my
memory start growing proportionally to the size of my index. In
detail, if I run "sar -r" i see that memory used doesn't go more than
12GB which is ok, but my kbcached continous growing. How can I limit
that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look
unnecessary or simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Just like any other JVM, best method is limiting the heap (ES uses an
environment variable $ES_HEAP_SIZE), and optionally, limit the direct
buffer memory with $ES_DIRECT_SIZE. Limiting the buffer decreases
performance and should not be necessary, because the OS will take really
good care of it, it will be distributed among the running processes at
demand.

See also bin/elasticsearch.in.sh for memory setup of the ES JVM startup
procedure.

Jörg

Am 20.03.13 19:17, schrieb Delmar:

I'm trying to understand, that's why I tried different aproaches.
My problem is that i want to limit the size of the cache because there
might be running other applications on the same server that might need
resources also. I don't want ES to own or try to own all the RAM
resources of my server. Nevertheless, is it possible to limit it? If
so, how can i do it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I've already limited the heap size to 4GB each, but not using the
environment variable $ES_HEAP_SIZE. I changed it directly in the
bin/elasticsearch script.
Regarding the $ES_DIRECT_SIZE, shouldn't my configurations on
conf/elasticsearch.yml
(namely the cache.memory.direct: false) or my bin/elasticsearch
configurations -XX:MaxDirectMemorySize have worked?
Nonetheless, I will give a try to your recommendations and see if it works.
Thanks.

2013/3/20 Jörg Prante joergprante@gmail.com

Just like any other JVM, best method is limiting the heap (ES uses an
environment variable $ES_HEAP_SIZE), and optionally, limit the direct
buffer memory with $ES_DIRECT_SIZE. Limiting the buffer decreases
performance and should not be necessary, because the OS will take really
good care of it, it will be distributed among the running processes at
demand.

See also bin/elasticsearch.in.sh for memory setup of the ES JVM startup
procedure.

Jörg

Am 20.03.13 19:17, schrieb Delmar:

I'm trying to understand, that's why I tried different aproaches.

My problem is that i want to limit the size of the cache because there
might be running other applications on the same server that might need
resources also. I don't want ES to own or try to own all the RAM resources
of my server. Nevertheless, is it possible to limit it? If so, how can i do
it?

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**Rt0Tej458ks/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Hugo Delmar Cabral

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

cache.memory.direct is just switching the JVM between using the direct
buffer memory of the OS (true) or the heap (false) for node-level
caching. Memory usage is the same. It is advisable to use the default
value which is true, so field cache elements are exposed very
efficiently to the garbage collection (they are not moved around).

Jörg

Am 20.03.13 19:52, schrieb Hugo Delmar Cabral:

I've already limited the heap size to 4GB each, but not using the
environment variable $ES_HEAP_SIZE. I changed it directly in the
bin/elasticsearch script.
Regarding the $ES_DIRECT_SIZE, shouldn't my configurations on
conf/elasticsearch.yml (namely the cache.memory.direct: false) or my
bin/elasticsearch configurations -XX:MaxDirectMemorySize have worked?
Nonetheless, I will give a try to your recommendations and see if it
works.
Thanks.

2013/3/20 Jörg Prante <joergprante@gmail.com
mailto:joergprante@gmail.com>

Just like any other JVM, best method is limiting the heap (ES uses
an environment variable $ES_HEAP_SIZE), and optionally, limit the
direct buffer memory with $ES_DIRECT_SIZE. Limiting the buffer
decreases performance and should not be necessary, because the OS
will take really good care of it, it will be distributed among the
running processes at demand.

See also bin/elasticsearch.in.sh <http://elasticsearch.in.sh> for
memory setup of the ES JVM startup procedure.

Jörg

Am 20.03.13 19:17, schrieb Delmar:

    I'm trying to understand, that's why I tried different aproaches.
    My problem is that i want to limit the size of the cache
    because there might be running other applications on the same
    server that might need resources also. I don't want ES to own
    or try to own all the RAM resources of my server.
    Nevertheless, is it possible to limit it? If so, how can i do it?


-- 
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Hugo Delmar Cabral

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ok, got it.
Unfortunately your previous suggestions didn't work! Do you have any other
suggestions?
The problem is not the heap size, it is the cache that continuous growing
accordingly to the size of my index until consumes all the available RAM of
the server!

2013/3/20 Jörg Prante joergprante@gmail.com

cache.memory.direct is just switching the JVM between using the direct
buffer memory of the OS (true) or the heap (false) for node-level caching.
Memory usage is the same. It is advisable to use the default value which is
true, so field cache elements are exposed very efficiently to the garbage
collection (they are not moved around).

Jörg

Am 20.03.13 19:52, schrieb Hugo Delmar Cabral:

I've already limited the heap size to 4GB each, but not using the
environment variable $ES_HEAP_SIZE. I changed it directly in the
bin/elasticsearch script.
Regarding the $ES_DIRECT_SIZE, shouldn't my configurations on
conf/elasticsearch.yml (namely the cache.memory.direct: false) or my
bin/elasticsearch configurations -XX:MaxDirectMemorySize have worked?
Nonetheless, I will give a try to your recommendations and see if it
works.
Thanks.

2013/3/20 Jörg Prante <joergprante@gmail.com <mailto:
joergprante@gmail.com>**>

Just like any other JVM, best method is limiting the heap (ES uses
an environment variable $ES_HEAP_SIZE), and optionally, limit the
direct buffer memory with $ES_DIRECT_SIZE. Limiting the buffer
decreases performance and should not be necessary, because the OS
will take really good care of it, it will be distributed among the
running processes at demand.

See also bin/elasticsearch.in.sh <http://elasticsearch.in.sh> for

memory setup of the ES JVM startup procedure.

Jörg

Am 20.03.13 19:17, schrieb Delmar:

    I'm trying to understand, that's why I tried different aproaches.
    My problem is that i want to limit the size of the cache
    because there might be running other applications on the same
    server that might need resources also. I don't want ES to own
    or try to own all the RAM resources of my server.
    Nevertheless, is it possible to limit it? If so, how can i do it?


--     You received this message because you are subscribed to a

topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
Rt0Tej458ks/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
<mailto:elasticsearch%**2Bunsubscribe@googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
**>.

For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>

.

--
Hugo Delmar Cabral

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**Rt0Tej458ks/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Hugo Delmar Cabral

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Well, the cache is supposed to cover all the RAM, as it's managed by the
OS, it must be assumed to be a correct decision. So what exactly are you
doing - do you send queries, do you index? What are the numbers that
worry you? Sorry but I can't figure it what the problem is.

Jörg

Am 20.03.13 20:49, schrieb Hugo Delmar Cabral:

The problem is not the heap size, it is the cache that continuous
growing accordingly to the size of my index until consumes all the
available RAM of the server!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I'm indexing, but since I'm trying to figure out the resources needed to
install an elastic search in a production environment, it's hard to do
figure out the needs in terms of RAM. Also, the usage or RAM/cache
influences the performance of the elastic search itself, regarding
throughput and latency. Summarizing, I'm trying to conduct performance and
sizing of ES, but it becomes hard when you don't control some variables...

2013/3/20 Jörg Prante joergprante@gmail.com

Well, the cache is supposed to cover all the RAM, as it's managed by the
OS, it must be assumed to be a correct decision. So what exactly are you
doing - do you send queries, do you index? What are the numbers that worry
you? Sorry but I can't figure it what the problem is.

Jörg

Am 20.03.13 20:49, schrieb Hugo Delmar Cabral:

The problem is not the heap size, it is the cache that continuous growing

accordingly to the size of my index until consumes all the available RAM of
the server!

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**Rt0Tej458ks/unsubscribe?hl=en-**UShttps://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Hugo Delmar Cabral

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Any other thoughts on this guys?
Thanks.

Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu:

Hi,

I'm having troubles managing the memory on my ES server. I have 3 nodes
with heap size 4GB each. However, when i start indexing my memory start
growing proportionally to the size of my index. In detail, if I run "sar
-r" i see that memory used doesn't go more than 12GB which is ok, but
my kbcached continous growing. How can I limit that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look unnecessary
or simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Are you simply trying to determine a "worst case" performance for ES?

It's unclear to me why you would want to limit the file OS cache (Direct
Buffer). The underlying OS will do a good job managing, freely giving
memory to new processes that need them (e.g. a new Apache process).

If memory contention with other processes is a serious concern (e.g.
running ES on the same host as an busy web stack) and you are worried about
performance, it is advisable to give ES it's own dedicated box. This is
similar advice for any data store - it is often preferable to run the
datastore/database on a dedicated machine to fully take advantage of
memory, reduce context switching, etc.

-Zach

On Thursday, March 21, 2013 10:52:17 AM UTC-4, Delmar wrote:

Any other thoughts on this guys?
Thanks.

Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu:

Hi,

I'm having troubles managing the memory on my ES server. I have 3 nodes
with heap size 4GB each. However, when i start indexing my memory start
growing proportionally to the size of my index. In detail, if I run "sar
-r" i see that memory used doesn't go more than 12GB which is ok, but
my kbcached continous growing. How can I limit that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look unnecessary
or simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I think you care for the wrong part of the system. You can control
indexing throughput, but not by intentionally limiting RAM memory.

Indexing in RAM is very fast, and the bottleneck are the disks. Even the
fastest SSD disk subsystem is 1000x slower than memory. Check if your
disks are fast enough for huge peaks of writes caused by segment
merging, this will limit your overall performance. Slow disk IO can be
compensated by store throttling described here (see "Store level
throttling"):
http://www.elasticsearch.org/guide/reference/index-modules/store.html

Jörg

Am 21.03.13 11:16, schrieb Hugo Delmar Cabral:

Also, the usage or RAM/cache influences the performance of the elastic
search itself, regarding throughput and latency. Summarizing, I'm
trying to conduct performance and sizing of ES, but it becomes hard
when you don't control some variables...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks guys. I understand and agree with your thoughts.
My question was to understand if we could control it or not! Because for my
tests it would be helpful to understand the impact of costs in RAM vs. gain
in throughput and latency.
But considering your answers I believe this is out of user control, meaning
that it is internally managed by the ES/OS.

Cheers

Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu:

Hi,

I'm having troubles managing the memory on my ES server. I have 3 nodes
with heap size 4GB each. However, when i start indexing my memory start
growing proportionally to the size of my index. In detail, if I run "sar
-r" i see that memory used doesn't go more than 12GB which is ok, but
my kbcached continous growing. How can I limit that?

I've tried some stuff, but since I'm newbie on this I must be missing
something here.
Here an example of my configurations by now (some might look unnecessary
or simple despair...) :slight_smile:

index.translog.flush_threshold_ops: 20000
index.store.type: niofs
index.store.fs.memory.enabled: false
index.cache.field.type: soft
cache.memory.direct: false
gateway.type: local

Can anyone help me please?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.