I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.
More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:
Watch out for the field data section in the indices stats.
A couple of comments about your settings:
Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.
Change the "indices.fielddata.cache.size: 2%" to something higher, like
"indices.fielddata.cache.size: 50%" and see how it goes from there.
Was there any reason for you change these? The defaults are usually good to
go with.
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%
Let me know how it goes.
Boaz
On Friday, October 11, 2013 12:48:41 PM UTC+2, subin wrote:
I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.
More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:
Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?
Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.
Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?
cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.
Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.
Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.
Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
Subin
*
*
On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:
Hi Subin,
Just leave it at it's default.
The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.
Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
hmm.. I'm experiencing build up of queue in redis. ie the size of the redis
process would usually be limited to 2-4Mb. But it has grown to 390Mb in teh
past couple of minutes. WHen I checked the queue usign LLEN logstash:redis,
it is 1052890. Normally it never goes beyond 500 when checked every second.
DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?
I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.
DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?
I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.
DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?
I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.
DOes this mean that ES is not indexing stuff fast enough and
therefore logstash is unable to pull from the queue and output to ES at the
rate logs are coming in?
I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?
The soft setting is a bad advice as you are much better off
planning and measuring what you need rather then overload the GC. A side
effect of the soft setting is that it will use all memory available,
potentially explaining what you see. I would suggest to change the setting
(remove the soft line and change the cache limit) and then based on what
you see decide on appropriate settings.
cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?
I think you got a misunderstanding from the quoted thread.
'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.
There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.
Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%
I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?
Thanks.
Re,
--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearch+unsubscribe@googlegroups.com.
Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.
I've increased my "indices.fielddata.cache.size" from 4 to 30%. No issues
yet. So moving on to Oracle Java 7, disabling "index.fielddata.cache: soft"
(removing it altogether) and increasing caching from 4 to 30% made
improvements for me. I have also increased "index.refresh_interval" to
120s. Earlier it was 10s.
Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
