Yet another memory usage issue - ES 0.90.5


(subin) #1

I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.

More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:

ES_HEAP_SIZE=4800m
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
index.number_of_shards: 1
index.number_of_replicas: 0
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%
index.fielddata.cache: soft
indices.fielddata.cache.size: 2%
bootstrap.mlockall: true

I also have set mappings for the indexes -

This is only a single ES server and no cluster. The total system RAM is 8G.

  • How does field and filter caches work?
  • Are there any additional performance configs I should consider?

Please advise.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #2

Hi Subin,

To find out what's causing the memory spike you can use the node
statistics: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html

Watch out for the field data section in the indices stats.

A couple of comments about your settings:

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.
Change the "indices.fielddata.cache.size: 2%" to something higher, like
"indices.fielddata.cache.size: 50%" and see how it goes from there.

Was there any reason for you change these? The defaults are usually good to
go with.
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%

Let me know how it goes.

Boaz

On Friday, October 11, 2013 12:48:41 PM UTC+2, subin wrote:

I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are
rotated daily and each day's index would amount to 12G. I'm planning to
retain 20 days of indexes, therefore at any time ES would maintain an index
of more or less 240G of indexers. As of now the total index size is only
58G.

More often than not, when a search is issued (over say, last 5 days
indexes) I see ES's "used heap memory" shoot up and remain almost like 500M
less than the "committed heap memory" which is 4.6G. Is it because of the
caches building up? I'm not sure as to how the filter and field cache
builds up here. What I'm worried is, that when our support team start using
it there will be more simultaneous queries to ES (through Kibana 3) and
this might lead to exhaustion of the Java Heap Mem. My current configs are
as:

ES_HEAP_SIZE=4800m
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
index.number_of_shards: 1
index.number_of_replicas: 0
index.translog.flush_threshold_ops: 2000
indices.memory.index_buffer_size: 40%
index.fielddata.cache: soft
indices.fielddata.cache.size: 2%
bootstrap.mlockall: true

I also have set mappings for the indexes -
https://gist.github.com/osssubb/6774727

This is only a single ES server and no cluster. The total system RAM is
8G.

  • How does field and filter caches work?
  • Are there any additional performance configs I should consider?

Please advise.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #3

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #4

Also, I got references for setting cache to 'soft' from
http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/..

thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 7:31 PM, subin ks ksubins321@gmail.com wrote:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #5

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on the
GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog setting
has anything to do with the number of documents handled by ES per sec? Is
it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #6

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually good
to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #7

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #8

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #9

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.
Thanks :slight_smile:


Subin
*
*
On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used up
quickly. In that case I guess it's more advisable to retain using 'soft'?
Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for example,
more memory for the cache, or a redesign of the app to use less cache
memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load on
the GC and doesn't allow you to control how much you want allocated to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #10

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning and
measuring what you need rather then overload the GC. A side effect of the
soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a load
on the GC and doesn't allow you to control how much you want allocated to
it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #11

Hi subin,

Unless you are experiencing problems you can leave it at the default (5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.com wrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates the
decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are usually
good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #12

hmm.. I'm experiencing build up of queue in redis. ie the size of the redis
process would usually be limited to 2-4Mb. But it has grown to 390Mb in teh
past couple of minutes. WHen I checked the queue usign LLEN logstash:redis,
it is 1052890. Normally it never goes beyond 500 when checked every second.

  1. DOes this mean that ES is not indexing stuff fast enough and therefore
    logstash is unable to pull from the queue and output to ES at the rate logs
    are coming in?

  2. Currently my index.refresh_interval is 10 seconds. Should I consider
    increasing it? maybe to 120s? I read from here that it can increase ES
    indexing speed -
    http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/

Please have an enlightment on 1 and 2.

Thanks Boaz.


Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.com wrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always used
up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #13

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks


Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

  1. DOes this mean that ES is not indexing stuff fast enough and therefore
    logstash is unable to pull from the queue and output to ES at the rate logs
    are coming in?

  2. Currently my index.refresh_interval is 10 seconds. Should I consider
    increasing it? maybe to 120s? I read from here that it can increase ES
    indexing speed -
    http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/

Please have an enlightment on 1 and 2.

Thanks Boaz.


Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into ES
using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.com wrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the translog
setting has anything to do with the number of documents handled by ES per
sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #14

I think you have another issue, what JVM version do you use?

Jörg
Am 16.10.2013 18:19 schrieb "subin ks" ksubins321@gmail.com:

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks


Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

  1. DOes this mean that ES is not indexing stuff fast enough and therefore
    logstash is unable to pull from the queue and output to ES at the rate logs
    are coming in?

  2. Currently my index.refresh_interval is 10 seconds. Should I consider
    increasing it? maybe to 120s? I read from here that it can increase ES
    indexing speed -
    http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/

Please have an enlightment on 1 and 2.

Thanks Boaz.


Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.comwrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off planning
and measuring what you need rather then overload the GC. A side effect of
the soft setting is that it will use all memory available, potentially
explaining what you see. I would suggest to change the setting (remove the
soft line and change the cache limit) and then based on what you see decide
on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #15

Hello Jorg,

I'm on Debian 6 with openjdk-6-jre.

Re,


Subin
*
*

On Wed, Oct 16, 2013 at 11:10 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you have another issue, what JVM version do you use?

Jörg
Am 16.10.2013 18:19 schrieb "subin ks" ksubins321@gmail.com:

Hello...

As an additional info I'm attaching jvm.png and jvm2.png which is a
snapshot of bigdesk of my current/live ES setup in this situation..

Thanks


Subin
*
*

On Wed, Oct 16, 2013 at 9:43 PM, subin ks ksubins321@gmail.com wrote:

hmm.. I'm experiencing build up of queue in redis. ie the size of the
redis process would usually be limited to 2-4Mb. But it has grown to 390Mb
in teh past couple of minutes. WHen I checked the queue usign LLEN
logstash:redis, it is 1052890. Normally it never goes beyond 500 when
checked every second.

  1. DOes this mean that ES is not indexing stuff fast enough and
    therefore logstash is unable to pull from the queue and output to ES at the
    rate logs are coming in?

  2. Currently my index.refresh_interval is 10 seconds. Should I consider
    increasing it? maybe to 120s? I read from here that it can increase ES
    indexing speed -
    http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/

Please have an enlightment on 1 and 2.

Thanks Boaz.


Subin
*
*

On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes b.leskes@gmail.com wrote:

Hi subin,

Unless you are experiencing problems you can leave it at the default
(5000).

Cheers,
Boaz

On Wed, Oct 16, 2013 at 5:48 PM, subin ks ksubins321@gmail.com wrote:

Hi,

Few doubts...

I just calculated that logstash feeds almost 1000 log-events/sec into
ES using elasticsearch_http output. What would be an advisable value
for index.translog.flush_threshold_ops in this case?

Thanks,


Subin
*
*

On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes b.leskes@gmail.comwrote:

Cool :wink:

Don't forget to increase the limit of the cache size.

On Tue, Oct 15, 2013 at 7:04 PM, subin ks ksubins321@gmail.comwrote:

I'll try with 'soft' setting removed.

Thanks :slight_smile:


Subin
*
*

On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes b.leskes@gmail.comwrote:

Hi Subin,

Just leave it at it's default.

The soft setting is a bad advice as you are much better off
planning and measuring what you need rather then overload the GC. A side
effect of the soft setting is that it will use all memory available,
potentially explaining what you see. I would suggest to change the setting
(remove the soft line and change the cache limit) and then based on what
you see decide on appropriate settings.

Cheers,
Boaz

On Tue, Oct 15, 2013 at 4:54 PM, subin ks ksubins321@gmail.comwrote:

cpu-load has been fine all throughout. It's the RAM which always
used up quickly. In that case I guess it's more advisable to retain using
'soft'? Or should I comment out the "index.fielddata.cache: soft" from the
configuration and leave it to the default?

Re,


Subin
*
*

On Tue, Oct 15, 2013 at 7:57 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I think you got a misunderstanding from the quoted thread.

'Soft' referencing does not prevent OOM at all. It just delegates
the decision about memory management of the cache to the Garbage Collector.
This means extra load (and often unpredictable load) on the CPU. If GC
fails, it will still OOM.

There are other and better methods to increase efficiency, for
example, more memory for the cache, or a redesign of the app to use less
cache memory.

Jörg
Am 15.10.2013 16:01 schrieb "subin ks" ksubins321@gmail.com:

Hi Boaz,

Remove the "index.fielddata.cache: soft" line. It only puts a
load on the GC and doesn't allow you to control how much you want allocated
to it.

I decided to set it to soft owing to suggestions from the group
herehttps://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14.
As said, it would help to prevent outofmemory issues. Therefore I had set
it to soft. I'd like to know your opinion too :slight_smile:

Was there any reason for you change these? The defaults are
usually good to go with.
index.translog.flush_**threshold_ops: 2000
indices.memory.index_buffer_**size: 40%

I'll set the above to defaults and let you know. Does the
translog setting has anything to do with the number of documents handled by
ES per sec? Is it set according to that basis?

Thanks.

Re,

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to a topic
in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #16

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK 6,
it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #17

Hello Jorg,

Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.

Thanks,


Subin
*
*

On Thu, Oct 17, 2013 at 12:37 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK
6, it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #18

I've increased my "indices.fielddata.cache.size" from 4 to 30%. No issues
yet. So moving on to Oracle Java 7, disabling "index.fielddata.cache: soft"
(removing it altogether) and increasing caching from 4 to 30% made
improvements for me. I have also increased "index.refresh_interval" to
120s. Earlier it was 10s.

Thanks to Boaz and Jorg.


Subin

On Tue, Oct 22, 2013 at 11:13 AM, subin ks ksubins321@gmail.com wrote:

Hello Jorg,

Thanks for the suggestion. Seeing memory improvements by using Oracle Java
7. But I withdrew 1 log-forwarding server from the cluster. I've avoided
"index.fielddata.cache: soft", infact I removed it. I have a few doubts
about how caches (index and field) works in ES. I'll raise a separate
thread for it.

Thanks,


Subin

On Thu, Oct 17, 2013 at 12:37 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK
6, it is known to have some odd memory behavior.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #19