Yet another memory usage issue - ES 0.90.5

subin · October 11, 2013, 10:48am

I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.

More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:

ES_HEAP_SIZE=4800m
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
index.number_of_shards: 1
index.number_of_replicas: 0
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%
index.fielddata.cache: soft
indices.fielddata.cache.size: 2%
bootstrap.mlockall: true

I also have set mappings for the indexes -

gist.github.com

https://gist.github.com/osssubb/6774727

gistfile1.txt

curl -XPUT http://10.0.4.24:9200/_template/logstash -d '
{
"template" : "logstash*",
"settings" : {
"number_of_shards" : 1,
"index.refresh_interval" : "10s",
"index.query.default_field": "@message"},
"mappings": {
"_default_": {
"_all": { "enabled": false },

This file has been truncated. show original

This is only a single ES server and no cluster. The total system RAM is 8G.

How does field and filter caches work?
Are there any additional performance configs I should consider?

Please advise.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · October 14, 2013, 2:46pm

Hi Subin,

To find out what's causing the memory spike you can use the node
statistics: Elasticsearch Platform — Find real-time answers at scale | Elastic

Watch out for the field data section in the indices stats.

A couple of comments about your settings:

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.
Change the "indices.fielddata.cache.size: 2%" to something higher, like
"indices.fielddata.cache.size: 50%" and see how it goes from there.

Was there any reason for you change these? The defaults are usually good to
go with.
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%

Let me know how it goes.

Boaz

On Friday, October 11, 2013 12:48:41 PM UTC+2, subin wrote:

I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.

More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:

ES_HEAP_SIZE=4800m
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
index.number_of_shards: 1
index.number_of_replicas: 0
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%
index.fielddata.cache: soft
indices.fielddata.cache.size: 2%
bootstrap.mlockall: true

I also have set mappings for the indexes -
@fields separated, date analyzed and 2 time-formats · GitHub

This is only a single ES server and no cluster. The total system RAM is
8G.

How does field and filter caches work?

Are there any additional performance configs I should consider?

Please advise.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 15, 2013, 2:01pm

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 15, 2013, 2:19pm

Also, I got references for setting cache to 'soft' from
ElasticSearch Cache Usage - Sematext..

thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 7:31 PM, subin ks ksubins321@gmail.com wrote:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 15, 2013, 2:27pm

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 15, 2013, 2:54pm

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · October 15, 2013, 3:34pm

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 15, 2013, 5:03pm

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · October 15, 2013, 5:12pm

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.
Thanks

Subin
*
*
On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 16, 2013, 3:48pm

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load
on the GC and doesn't allow you to control how much you want allocated to
it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Boaz_Leskes · October 16, 2013, 4:02pm

Hi subin,

Unless you are experiencing problems you can leave it at the default (5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 16, 2013, 4:13pm

hmm.. I'm experiencing build up of queue in redis. ie the size of the redis
process would usually be limited to 2-4Mb. But it has grown to 390Mb in teh
past couple of minutes. WHen I checked the queue usign LLEN logstash:redis,
it is 1052890. Normally it never goes beyond 500 when checked every second.

DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?
Currently my index.refresh_interval is 10 seconds. Should I consider
increasing it? maybe to 120s? I read from here that it can increase ES
indexing speed -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Please have an enlightment on 1 and 2.

Thanks Boaz.

Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 16, 2013, 4:19pm

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks

Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?

Currently my index.refresh_interval is 10 seconds. Should I consider
increasing it? maybe to 120s? I read from here that it can increase ES
indexing speed -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Please have an enlightment on 1 and 2.

Thanks Boaz.

Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 16, 2013, 5:40pm

I think you have another issue, what JVM version do you use?

Jörg
Am 16.10.2013 18:19 schrieb "subin ks" ksubins321@gmail.com:

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks

Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

DOes this mean that ES is not indexing stuff fast enough and therefore
logstash is unable to pull from the queue and output to ES at the rate logs
are coming in?

Currently my index.refresh_interval is 10 seconds. Should I consider
increasing it? maybe to 120s? I read from here that it can increase ES
indexing speed -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Please have an enlightment on 1 and 2.

Thanks Boaz.

Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.comwrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 16, 2013, 5:49pm

Hello Jorg,

I'm on Debian 6 with openjdk-6-jre.

Re,

Subin
*
*

On Wed, Oct 16, 2013 at 11:10 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you have another issue, what JVM version do you use?

Jörg
Am 16.10.2013 18:19 schrieb "subin ks" ksubins321@gmail.com:

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks

Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

DOes this mean that ES is not indexing stuff fast enough and
therefore logstash is unable to pull from the queue and output to ES at the
rate logs are coming in?

Currently my index.refresh_interval is 10 seconds. Should I consider
increasing it? maybe to 120s? I read from here that it can increase ES
indexing speed -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Please have an enlightment on 1 and 2.

Thanks Boaz.

Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,

Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.comwrote:

I'll try with 'soft' setting removed.

Thanks

Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off
planning and measuring what you need rather then overload the GC. A side
effect of the soft setting is that it will use all memory available,
potentially explaining what you see. I would suggest to change the setting
(remove the soft line and change the cache limit) and then based on what
you see decide on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,

Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to a topic
in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 16, 2013, 7:07pm

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK 6,
it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · October 22, 2013, 5:43am

Hello Jorg,

Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.

Thanks,

Subin
*
*

On Thu, Oct 17, 2013 at 12:37 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK
6, it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

subin · November 11, 2013, 2:56pm

I've increased my "indices.fielddata.cache.size" from 4 to 30%. No issues
yet. So moving on to Oracle Java 7, disabling "index.fielddata.cache: soft"
(removing it altogether) and increasing caching from 4 to 30% made
improvements for me. I have also increased "index.refresh_interval" to
120s. Earlier it was 10s.

Thanks to Boaz and Jorg.

Subin

On Tue, Oct 22, 2013 at 11:13 AM, subin ks ksubins321@gmail.com wrote:

Hello Jorg,

Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.

Thanks,

Subin

On Thu, Oct 17, 2013 at 12:37 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK
6, it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
ES seen to allocate more memory outside ES_HEAP_SIZE using "top" Elasticsearch	3	804	July 6, 2017
Elasticsearch 1.5.2 deployment issue Elasticsearch	4	542	July 5, 2017
35 shards but maxing out JVM heap Elasticsearch	12	4037	April 5, 2018
Downward spiral of free memory and too many ES requests Elasticsearch	13	587	July 6, 2017
ElasticSearch out of memory with static field index cache size Elasticsearch	6	820	July 6, 2017

Yet another memory usage issue - ES 0.90.5

Related topics