# Yet another memory usage issue - ES 0.90.5 **URL:** \ **Category:** Elasticsearch\ **Created:** [October 11, 2013, 10:48am UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922 "2013-10-11T10:48:41Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 11, 2013, 10:48am UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/1 "2013-10-11T10:48:41Z")
I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are rotated daily and each day's index would amount to 12G. I'm planning to retain 20 days of indexes, therefore at any time ES would maintain an index of more or less 240G of indexers. As of now the total index size is only 58G. More often than not, when a search is issued (over say, last 5 days indexes) I see ES's "used heap memory" shoot up and remain almost like 500M less than the "committed heap memory" which is 4.6G. Is it because of the caches building up? I'm not sure as to how the filter and field cache builds up here. What I'm worried is, that when our support team start using it there will be more simultaneous queries to ES (through Kibana 3) and this might lead to exhaustion of the Java Heap Mem. My current configs are as: ES\_HEAP\_SIZE=4800m MAX\_OPEN\_FILES=65535 MAX\_LOCKED\_MEMORY=unlimited index.number\_of\_shards: 1 index.number\_of\_replicas: 0 index.translog.flush\_threshold\_ops: 2000 indices.memory.index\_buffer\_size: 40% index.fielddata.cache: soft indices.fielddata.cache.size: 2% bootstrap.mlockall: true I also have set mappings for the indexes - > This is only a single ES server and no cluster. The total system RAM is 8G. - How does field and filter caches work? - Are there any additional performance configs I should consider? Please advise. Thanks, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 14, 2013, 2:46pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/2 "2013-10-14T14:46:16Z")
Hi Subin, To find out what's causing the memory spike you can use the node statistics: [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html) Watch out for the field data section in the indices stats. A couple of comments about your settings: Remove the "index.fielddata.cache: soft" line. It only puts a load on the GC and doesn't allow you to control how much you want allocated to it. Change the "indices.fielddata.cache.size: 2%" to something higher, like "indices.fielddata.cache.size: 50%" and see how it goes from there. Was there any reason for you change these? The defaults are usually good to go with. index.translog.flush\_threshold\_ops: 2000 indices.memory.index\_buffer\_size: 40% Let me know how it goes. Boaz On Friday, October 11, 2013 12:48:41 PM UTC+2, subin wrote: > I have ES 0.90.5 in my Logstash setup (Debian 6). I have indices which are > rotated daily and each day's index would amount to 12G. I'm planning to > retain 20 days of indexes, therefore at any time ES would maintain an index > of more or less 240G of indexers. As of now the total index size is only > 58G. > > More often than not, when a search is issued (over say, last 5 days > indexes) I see ES's "used heap memory" shoot up and remain almost like 500M > less than the "committed heap memory" which is 4.6G. Is it because of the > caches building up? I'm not sure as to how the filter and field cache > builds up here. What I'm worried is, that when our support team start using > it there will be more simultaneous queries to ES (through Kibana 3) and > this might lead to exhaustion of the Java Heap Mem. My current configs are > as: > > ES\_HEAP\_SIZE=4800m > MAX\_OPEN\_FILES=65535 > MAX\_LOCKED\_MEMORY=unlimited > index.number\_of\_shards: 1 > index.number\_of\_replicas: 0 > index.translog.flush\_threshold\_ops: 2000 > indices.memory.index\_buffer\_size: 40% > index.fielddata.cache: soft > indices.fielddata.cache.size: 2% > bootstrap.mlockall: true > > I also have set mappings for the indexes - > [@fields separated, date analyzed and 2 time-formats ยท GitHub](https://gist.github.com/osssubb/6774727) > > This is only a single ES server and no cluster. The total system RAM is > 8G. > > - How does field and filter caches work? > - Are there any additional performance configs I should consider? > > Please advise. > > Thanks, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 15, 2013, 2:01pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/3 "2013-10-15T14:01:40Z")
Hi Boaz, > Remove the "index.fielddata.cache: soft" line. It only puts a load on the > GC and doesn't allow you to control how much you want allocated to it. I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). As said, it would help to prevent outofmemory issues. Therefore I had set it to soft. I'd like to know your opinion too ๐Ÿ™‚ > Was there any reason for you change these? The defaults are usually good > to go with. > index.translog.flush\_\*\*threshold\_ops: 2000 > indices.memory.index\_buffer\_\*\*size: 40% I'll set the above to defaults and let you know. Does the translog setting has anything to do with the number of documents handled by ES per sec? Is it set according to that basis? Thanks. Re, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 15, 2013, 2:19pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/4 "2013-10-15T14:19:52Z")
Also, I got references for setting cache to 'soft' from [ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/).. thanks, * * * Subin \* \* On Tue, Oct 15, 2013 at 7:31 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > Hi Boaz, > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on the > > GC and doesn't allow you to control how much you want allocated to it. > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > As said, it would help to prevent outofmemory issues. Therefore I had set > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > Was there any reason for you change these? The defaults are usually good > > to go with. > > index.translog.flush\_\*\*threshold\_ops: 2000 > > indices.memory.index\_buffer\_\*\*size: 40% > > I'll set the above to defaults and let you know. Does the translog setting > has anything to do with the number of documents handled by ES per sec? Is > it set according to that basis? > > Thanks. > > Re, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 15, 2013, 2:27pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/5 "2013-10-15T14:27:38Z")
I think you got a misunderstanding from the quoted thread. 'Soft' referencing does not prevent OOM at all. It just delegates the decision about memory management of the cache to the Garbage Collector. This means extra load (and often unpredictable load) on the CPU. If GC fails, it will still OOM. There are other and better methods to increase efficiency, for example, more memory for the cache, or a redesign of the app to use less cache memory. Jรถrg Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > Hi Boaz, > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on the > > GC and doesn't allow you to control how much you want allocated to it. > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > As said, it would help to prevent outofmemory issues. Therefore I had set > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > Was there any reason for you change these? The defaults are usually good > > to go with. > > index.translog.flush\_\*\*threshold\_ops: 2000 > > indices.memory.index\_buffer\_\*\*size: 40% > > I'll set the above to defaults and let you know. Does the translog setting > has anything to do with the number of documents handled by ES per sec? Is > it set according to that basis? > > Thanks. > > Re, > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 15, 2013, 2:54pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/6 "2013-10-15T14:54:41Z")
cpu-load has been fine all throughout. It's the RAM which always used up quickly. In that case I guess it's more advisable to retain using 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the configuration and leave it to the default? Re, * * * Subin \* \* On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I think you got a misunderstanding from the quoted thread. > > 'Soft' referencing does not prevent OOM at all. It just delegates the > decision about memory management of the cache to the Garbage Collector. > This means extra load (and often unpredictable load) on the CPU. If GC > fails, it will still OOM. > > There are other and better methods to increase efficiency, for example, > more memory for the cache, or a redesign of the app to use less cache > memory. > > Jรถrg > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > Hi Boaz, > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on > > > the GC and doesn't allow you to control how much you want allocated to it. > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > As said, it would help to prevent outofmemory issues. Therefore I had set > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > Was there any reason for you change these? The defaults are usually good > > > to go with. > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > I'll set the above to defaults and let you know. Does the translog > > setting has anything to do with the number of documents handled by ES per > > sec? Is it set according to that basis? > > > > Thanks. > > > > Re, > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 15, 2013, 3:34pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/7 "2013-10-15T15:34:36Z")
Hi Subin, Just leave it at it's default. The soft setting is a bad advice as you are much better off planning and measuring what you need rather then overload the GC. A side effect of the soft setting is that it will use all memory available, potentially explaining what you see. I would suggest to change the setting (remove the soft line and change the cache limit) and then based on what you see decide on appropriate settings. Cheers, Boaz On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > cpu-load has been fine all throughout. It's the RAM which always used up > quickly. In that case I guess it's more advisable to retain using 'soft'? > Or should I comment out the "index.fielddata.cache: soft" from the > configuration and leave it to the default? > > Re, > > * * * > > Subin > \* > \* > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > I think you got a misunderstanding from the quoted thread. > > > > 'Soft' referencing does not prevent OOM at all. It just delegates the > > decision about memory management of the cache to the Garbage Collector. > > This means extra load (and often unpredictable load) on the CPU. If GC > > fails, it will still OOM. > > > > There are other and better methods to increase efficiency, for example, > > more memory for the cache, or a redesign of the app to use less cache > > memory. > > > > Jรถrg > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > Hi Boaz, > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on > > > > the GC and doesn't allow you to control how much you want allocated to it. > > > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > Was there any reason for you change these? The defaults are usually > > > > good to go with. > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > I'll set the above to defaults and let you know. Does the translog > > > setting has anything to do with the number of documents handled by ES per > > > sec? Is it set according to that basis? > > > > > > Thanks. > > > > > > Re, > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 15, 2013, 5:03pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/8 "2013-10-15T17:03:53Z")
I'll try with 'soft' setting removed. Thanks ๐Ÿ™‚ * * * Subin \* \* On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > Hi Subin, > > Just leave it at it's default. > > The soft setting is a bad advice as you are much better off planning and > measuring what you need rather then overload the GC. A side effect of the > soft setting is that it will use all memory available, potentially > explaining what you see. I would suggest to change the setting (remove the > soft line and change the cache limit) and then based on what you see decide > on appropriate settings. > > Cheers, > Boaz > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > cpu-load has been fine all throughout. It's the RAM which always used up > > quickly. In that case I guess it's more advisable to retain using 'soft'? > > Or should I comment out the "index.fielddata.cache: soft" from the > > configuration and leave it to the default? > > > > Re, > > > > * * * > > > > Subin > > \* > > \* > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates the > > > decision about memory management of the cache to the Garbage Collector. > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > fails, it will still OOM. > > > > > > There are other and better methods to increase efficiency, for example, > > > more memory for the cache, or a redesign of the app to use less cache > > > memory. > > > > > > Jรถrg > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > Hi Boaz, > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on > > > > > the GC and doesn't allow you to control how much you want allocated to it. > > > > > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > Was there any reason for you change these? The defaults are usually > > > > > good to go with. > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > setting has anything to do with the number of documents handled by ES per > > > > sec? Is it set according to that basis? > > > > > > > > Thanks. > > > > > > > > Re, > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 15, 2013, 5:12pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/9 "2013-10-15T17:12:35Z")
Cool ๐Ÿ˜‰ Don't forget to increase the limit of the cache size. On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > I'll try with 'soft' setting removed. > Thanks ๐Ÿ™‚ > > * * * > > Subin > \* > \* > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > Hi Subin, > > > > Just leave it at it's default. > > > > The soft setting is a bad advice as you are much better off planning and > > measuring what you need rather then overload the GC. A side effect of the > > soft setting is that it will use all memory available, potentially > > explaining what you see. I would suggest to change the setting (remove the > > soft line and change the cache limit) and then based on what you see decide > > on appropriate settings. > > > > Cheers, > > Boaz > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > cpu-load has been fine all throughout. It's the RAM which always used up > > > quickly. In that case I guess it's more advisable to retain using 'soft'? > > > Or should I comment out the "index.fielddata.cache: soft" from the > > > configuration and leave it to the default? > > > > > > Re, > > > > > > * * * > > > > > > Subin > > > \* > > > \* > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates the > > > > decision about memory management of the cache to the Garbage Collector. > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > fails, it will still OOM. > > > > > > > > There are other and better methods to increase efficiency, for example, > > > > more memory for the cache, or a redesign of the app to use less cache > > > > memory. > > > > > > > > Jรถrg > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > Hi Boaz, > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a load on > > > > > > the GC and doesn't allow you to control how much you want allocated to it. > > > > > > > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > Was there any reason for you change these? The defaults are usually > > > > > > good to go with. > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > > setting has anything to do with the number of documents handled by ES per > > > > > sec? Is it set according to that basis? > > > > > > > > > > Thanks. > > > > > > > > > > Re, > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 16, 2013, 3:48pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/10 "2013-10-16T15:48:51Z")
Hi, Few doubts... I just calculated that logstash feeds almost 1000 log-events/sec into ES using elasticsearch\_http output. What would be an advisable value for index.translog.flush\_threshold\_ops in this case? Thanks, * * * Subin \* \* On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > Cool ๐Ÿ˜‰ > > Don't forget to increase the limit of the cache size. > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > I'll try with 'soft' setting removed. > > > > Thanks ๐Ÿ™‚ > > > > * * * > > > > Subin > > \* > > \* > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > > > Hi Subin, > > > > > > Just leave it at it's default. > > > > > > The soft setting is a bad advice as you are much better off planning and > > > measuring what you need rather then overload the GC. A side effect of the > > > soft setting is that it will use all memory available, potentially > > > explaining what you see. I would suggest to change the setting (remove the > > > soft line and change the cache limit) and then based on what you see decide > > > on appropriate settings. > > > > > > Cheers, > > > Boaz > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > cpu-load has been fine all throughout. It's the RAM which always used > > > > up quickly. In that case I guess it's more advisable to retain using > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > configuration and leave it to the default? > > > > > > > > Re, > > > > > > > > * * * > > > > > > > > Subin > > > > \* > > > > \* > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates the > > > > > decision about memory management of the cache to the Garbage Collector. > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > fails, it will still OOM. > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > cache memory. > > > > > > > > > > Jรถrg > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a load > > > > > > > on the GC and doesn't allow you to control how much you want allocated to > > > > > > > it. > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > Was there any reason for you change these? The defaults are usually > > > > > > > good to go with. > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > > > setting has anything to do with the number of documents handled by ES per > > > > > > sec? Is it set according to that basis? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Re, > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 16, 2013, 4:02pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/11 "2013-10-16T16:02:05Z")
Hi subin, Unless you are experiencing problems you can leave it at the default (5000). Cheers, Boaz On Wed, Oct 16, 2013 at 5:48 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > Hi, > > Few doubts... > > I just calculated that logstash feeds almost 1000 log-events/sec into ES > using elasticsearch\_http output. What would be an advisable value > for index.translog.flush\_threshold\_ops in this case? > > Thanks, > > * * * > > Subin > \* > \* > > On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > Cool ๐Ÿ˜‰ > > > > Don't forget to increase the limit of the cache size. > > > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > I'll try with 'soft' setting removed. > > > > > > Thanks ๐Ÿ™‚ > > > > > > * * * > > > > > > Subin > > > \* > > > \* > > > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > > > > > Hi Subin, > > > > > > > > Just leave it at it's default. > > > > > > > > The soft setting is a bad advice as you are much better off planning > > > > and measuring what you need rather then overload the GC. A side effect of > > > > the soft setting is that it will use all memory available, potentially > > > > explaining what you see. I would suggest to change the setting (remove the > > > > soft line and change the cache limit) and then based on what you see decide > > > > on appropriate settings. > > > > > > > > Cheers, > > > > Boaz > > > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > > > cpu-load has been fine all throughout. It's the RAM which always used > > > > > up quickly. In that case I guess it's more advisable to retain using > > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > > configuration and leave it to the default? > > > > > > > > > > Re, > > > > > > > > > > * * * > > > > > > > > > > Subin > > > > > \* > > > > > \* > > > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates the > > > > > > decision about memory management of the cache to the Garbage Collector. > > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > > fails, it will still OOM. > > > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > > cache memory. > > > > > > > > > > > > Jรถrg > > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a > > > > > > > > load on the GC and doesn't allow you to control how much you want allocated > > > > > > > > to it. > > > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > > > Was there any reason for you change these? The defaults are usually > > > > > > > > good to go with. > > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > > > > setting has anything to do with the number of documents handled by ES per > > > > > > > sec? Is it set according to that basis? > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 16, 2013, 4:13pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/12 "2013-10-16T16:13:20Z")
hmm.. I'm experiencing build up of queue in redis. ie the size of the redis process would usually be limited to 2-4Mb. But it has grown to 390Mb in teh past couple of minutes. WHen I checked the queue usign LLEN logstash:redis, it is 1052890. Normally it never goes beyond 500 when checked every second. 1. DOes this mean that ES is not indexing stuff fast enough and therefore logstash is unable to pull from the queue and output to ES at the rate logs are coming in? 2. Currently my index.refresh\_interval is 10 seconds. Should I consider increasing it? maybe to 120s? I read from here that it can increase ES indexing speed - [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/) Please have an enlightment on 1 and 2. Thanks Boaz. * * * Subin \* \* On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > Hi subin, > > Unless you are experiencing problems you can leave it at the default > (5000). > > Cheers, > Boaz > > On Wed, Oct 16, 2013 at 5:48 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > Hi, > > > > Few doubts... > > > > I just calculated that logstash feeds almost 1000 log-events/sec into ES > > using elasticsearch\_http output. What would be an advisable value > > for index.translog.flush\_threshold\_ops in this case? > > > > Thanks, > > > > * * * > > > > Subin > > \* > > \* > > > > On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > > > Cool ๐Ÿ˜‰ > > > > > > Don't forget to increase the limit of the cache size. > > > > > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > I'll try with 'soft' setting removed. > > > > > > > > Thanks ๐Ÿ™‚ > > > > > > > > * * * > > > > > > > > Subin > > > > \* > > > > \* > > > > > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > Hi Subin, > > > > > > > > > > Just leave it at it's default. > > > > > > > > > > The soft setting is a bad advice as you are much better off planning > > > > > and measuring what you need rather then overload the GC. A side effect of > > > > > the soft setting is that it will use all memory available, potentially > > > > > explaining what you see. I would suggest to change the setting (remove the > > > > > soft line and change the cache limit) and then based on what you see decide > > > > > on appropriate settings. > > > > > > > > > > Cheers, > > > > > Boaz > > > > > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > cpu-load has been fine all throughout. It's the RAM which always used > > > > > > up quickly. In that case I guess it's more advisable to retain using > > > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > > > configuration and leave it to the default? > > > > > > > > > > > > Re, > > > > > > > > > > > > * * * > > > > > > > > > > > > Subin > > > > > > \* > > > > > > \* > > > > > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates > > > > > > > the decision about memory management of the cache to the Garbage Collector. > > > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > > > fails, it will still OOM. > > > > > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > > > cache memory. > > > > > > > > > > > > > > Jรถrg > > > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a > > > > > > > > > load on the GC and doesn't allow you to control how much you want allocated > > > > > > > > > to it. > > > > > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group > > > > > > > > here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > > > > > Was there any reason for you change these? The defaults are > > > > > > > > > usually good to go with. > > > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > > > > > setting has anything to do with the number of documents handled by ES per > > > > > > > > sec? Is it set according to that basis? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 16, 2013, 4:19pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/13 "2013-10-16T16:19:21Z")
Hello... As an additional info I'm attaching jvm.png and jvm2.png which is a snapshot of bigdesk of my current/live ES setup in this situation.. Thanks * * * Subin \* \* On Wed, Oct 16, 2013 at 9:43 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > hmm.. I'm experiencing build up of queue in redis. ie the size of the > redis process would usually be limited to 2-4Mb. But it has grown to 390Mb > in teh past couple of minutes. WHen I checked the queue usign LLEN > logstash:redis, it is 1052890. Normally it never goes beyond 500 when > checked every second. > > 1. DOes this mean that ES is not indexing stuff fast enough and therefore > logstash is unable to pull from the queue and output to ES at the rate logs > are coming in? > > 2. Currently my index.refresh\_interval is 10 seconds. Should I consider > increasing it? maybe to 120s? I read from here that it can increase ES > indexing speed - > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/) > > Please have an enlightment on 1 and 2. > > Thanks Boaz. > > * * * > > Subin > \* > \* > > On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > Hi subin, > > > > Unless you are experiencing problems you can leave it at the default > > (5000). > > > > Cheers, > > Boaz > > > > On Wed, Oct 16, 2013 at 5:48 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > Hi, > > > > > > Few doubts... > > > > > > I just calculated that logstash feeds almost 1000 log-events/sec into ES > > > using elasticsearch\_http output. What would be an advisable value > > > for index.translog.flush\_threshold\_ops in this case? > > > > > > Thanks, > > > > > > * * * > > > > > > Subin > > > \* > > > \* > > > > > > On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > Cool ๐Ÿ˜‰ > > > > > > > > Don't forget to increase the limit of the cache size. > > > > > > > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > > > I'll try with 'soft' setting removed. > > > > > > > > > > Thanks ๐Ÿ™‚ > > > > > > > > > > * * * > > > > > > > > > > Subin > > > > > \* > > > > > \* > > > > > > > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > > > Hi Subin, > > > > > > > > > > > > Just leave it at it's default. > > > > > > > > > > > > The soft setting is a bad advice as you are much better off planning > > > > > > and measuring what you need rather then overload the GC. A side effect of > > > > > > the soft setting is that it will use all memory available, potentially > > > > > > explaining what you see. I would suggest to change the setting (remove the > > > > > > soft line and change the cache limit) and then based on what you see decide > > > > > > on appropriate settings. > > > > > > > > > > > > Cheers, > > > > > > Boaz > > > > > > > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > > > cpu-load has been fine all throughout. It's the RAM which always > > > > > > > used up quickly. In that case I guess it's more advisable to retain using > > > > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > > > > configuration and leave it to the default? > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > * * * > > > > > > > > > > > > > > Subin > > > > > > > \* > > > > > > > \* > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates > > > > > > > > the decision about memory management of the cache to the Garbage Collector. > > > > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > > > > fails, it will still OOM. > > > > > > > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > > > > cache memory. > > > > > > > > > > > > > > > > Jรถrg > > > > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a > > > > > > > > > > load on the GC and doesn't allow you to control how much you want allocated > > > > > > > > > > to it. > > > > > > > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group > > > > > > > > > here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > > > > > > > Was there any reason for you change these? The defaults are > > > > > > > > > > usually good to go with. > > > > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the translog > > > > > > > > > setting has anything to do with the number of documents handled by ES per > > > > > > > > > sec? Is it set according to that basis? > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 16, 2013, 5:40pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/14 "2013-10-16T17:40:26Z")
I think you have another issue, what JVM version do you use? Jรถrg Am 16.10.2013 18:19 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > Hello... > > As an additional info I'm attaching jvm.png and jvm2.png which is a > snapshot of bigdesk of my current/live ES setup in this situation.. > > Thanks > > * * * > > Subin > \* > \* > > On Wed, Oct 16, 2013 at 9:43 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > hmm.. I'm experiencing build up of queue in redis. ie the size of the > > redis process would usually be limited to 2-4Mb. But it has grown to 390Mb > > in teh past couple of minutes. WHen I checked the queue usign LLEN > > logstash:redis, it is 1052890. Normally it never goes beyond 500 when > > checked every second. > > > > 1. DOes this mean that ES is not indexing stuff fast enough and therefore > > logstash is unable to pull from the queue and output to ES at the rate logs > > are coming in? > > > > 2. Currently my index.refresh\_interval is 10 seconds. Should I consider > > increasing it? maybe to 120s? I read from here that it can increase ES > > indexing speed - > > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/) > > > > Please have an enlightment on 1 and 2. > > > > Thanks Boaz. > > > > * * * > > > > Subin > > \* > > \* > > > > On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > > > Hi subin, > > > > > > Unless you are experiencing problems you can leave it at the default > > > (5000). > > > > > > Cheers, > > > Boaz > > > > > > On Wed, Oct 16, 2013 at 5:48 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > Few doubts... > > > > > > > > I just calculated that logstash feeds almost 1000 log-events/sec into > > > > ES using elasticsearch\_http output. What would be an advisable value > > > > for index.translog.flush\_threshold\_ops in this case? > > > > > > > > Thanks, > > > > > > > > * * * > > > > > > > > Subin > > > > \* > > > > \* > > > > > > > > On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > Cool ๐Ÿ˜‰ > > > > > > > > > > Don't forget to increase the limit of the cache size. > > > > > > > > > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > I'll try with 'soft' setting removed. > > > > > > > > > > > > Thanks ๐Ÿ™‚ > > > > > > > > > > > > * * * > > > > > > > > > > > > Subin > > > > > > \* > > > > > > \* > > > > > > > > > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > > > > > Hi Subin, > > > > > > > > > > > > > > Just leave it at it's default. > > > > > > > > > > > > > > The soft setting is a bad advice as you are much better off planning > > > > > > > and measuring what you need rather then overload the GC. A side effect of > > > > > > > the soft setting is that it will use all memory available, potentially > > > > > > > explaining what you see. I would suggest to change the setting (remove the > > > > > > > soft line and change the cache limit) and then based on what you see decide > > > > > > > on appropriate settings. > > > > > > > > > > > > > > Cheers, > > > > > > > Boaz > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > > > > > cpu-load has been fine all throughout. It's the RAM which always > > > > > > > > used up quickly. In that case I guess it's more advisable to retain using > > > > > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > > > > > configuration and leave it to the default? > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > * * * > > > > > > > > > > > > > > > > Subin > > > > > > > > \* > > > > > > > > \* > > > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates > > > > > > > > > the decision about memory management of the cache to the Garbage Collector. > > > > > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > > > > > fails, it will still OOM. > > > > > > > > > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > > > > > cache memory. > > > > > > > > > > > > > > > > > > Jรถrg > > > > > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a > > > > > > > > > > > load on the GC and doesn't allow you to control how much you want allocated > > > > > > > > > > > to it. > > > > > > > > > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group > > > > > > > > > > here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > > > > > > > > > Was there any reason for you change these? The defaults are > > > > > > > > > > > usually good to go with. > > > > > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the > > > > > > > > > > translog setting has anything to do with the number of documents handled by > > > > > > > > > > ES per sec? Is it set according to that basis? > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > > . > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 16, 2013, 5:49pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/15 "2013-10-16T17:49:05Z")
Hello Jorg, I'm on Debian 6 with openjdk-6-jre. Re, * * * Subin \* \* On Wed, Oct 16, 2013 at 11:10 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I think you have another issue, what JVM version do you use? > > Jรถrg > Am 16.10.2013 18:19 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > Hello... > > > As an additional info I'm attaching jvm.png and jvm2.png which is a > > snapshot of bigdesk of my current/live ES setup in this situation.. > > > > Thanks > > > > * * * > > > > Subin > > \* > > \* > > > > On Wed, Oct 16, 2013 at 9:43 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > hmm.. I'm experiencing build up of queue in redis. ie the size of the > > > redis process would usually be limited to 2-4Mb. But it has grown to 390Mb > > > in teh past couple of minutes. WHen I checked the queue usign LLEN > > > logstash:redis, it is 1052890. Normally it never goes beyond 500 when > > > checked every second. > > > > > > 1. DOes this mean that ES is not indexing stuff fast enough and > > > therefore logstash is unable to pull from the queue and output to ES at the > > > rate logs are coming in? > > > > > > 2. Currently my index.refresh\_interval is 10 seconds. Should I consider > > > increasing it? maybe to 120s? I read from here that it can increase ES > > > indexing speed - > > > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/) > > > > > > Please have an enlightment on 1 and 2. > > > > > > Thanks Boaz. > > > > > > * * * > > > > > > Subin > > > \* > > > \* > > > > > > On Wed, Oct 16, 2013 at 9:32 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > > > > > > Hi subin, > > > > > > > > Unless you are experiencing problems you can leave it at the default > > > > (5000). > > > > > > > > Cheers, > > > > Boaz > > > > > > > > On Wed, Oct 16, 2013 at 5:48 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > Few doubts... > > > > > > > > > > I just calculated that logstash feeds almost 1000 log-events/sec into > > > > > ES using elasticsearch\_http output. What would be an advisable value > > > > > for index.translog.flush\_threshold\_ops in this case? > > > > > > > > > > Thanks, > > > > > > > > > > * * * > > > > > > > > > > Subin > > > > > \* > > > > > \* > > > > > > > > > > On Tue, Oct 15, 2013 at 10:42 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > > > Cool ๐Ÿ˜‰ > > > > > > > > > > > > Don't forget to increase the limit of the cache size. > > > > > > > > > > > > On Tue, Oct 15, 2013 at 7:04 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > > > I'll try with 'soft' setting removed. > > > > > > > > > > > > > > Thanks ๐Ÿ™‚ > > > > > > > > > > > > > > * * * > > > > > > > > > > > > > > Subin > > > > > > > \* > > > > > > > \* > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 9:04 PM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com)wrote: > > > > > > > > > > > > > > > Hi Subin, > > > > > > > > > > > > > > > > Just leave it at it's default. > > > > > > > > > > > > > > > > The soft setting is a bad advice as you are much better off > > > > > > > > planning and measuring what you need rather then overload the GC. A side > > > > > > > > effect of the soft setting is that it will use all memory available, > > > > > > > > potentially explaining what you see. I would suggest to change the setting > > > > > > > > (remove the soft line and change the cache limit) and then based on what > > > > > > > > you see decide on appropriate settings. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Boaz > > > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 4:54 PM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com)wrote: > > > > > > > > > > > > > > > > > cpu-load has been fine all throughout. It's the RAM which always > > > > > > > > > used up quickly. In that case I guess it's more advisable to retain using > > > > > > > > > 'soft'? Or should I comment out the "index.fielddata.cache: soft" from the > > > > > > > > > configuration and leave it to the default? > > > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > > > * * * > > > > > > > > > > > > > > > > > > Subin > > > > > > > > > \* > > > > > > > > > \* > > > > > > > > > > > > > > > > > > On Tue, Oct 15, 2013 at 7:57 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > > > > > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > I think you got a misunderstanding from the quoted thread. > > > > > > > > > > > > > > > > > > > > 'Soft' referencing does not prevent OOM at all. It just delegates > > > > > > > > > > the decision about memory management of the cache to the Garbage Collector. > > > > > > > > > > This means extra load (and often unpredictable load) on the CPU. If GC > > > > > > > > > > fails, it will still OOM. > > > > > > > > > > > > > > > > > > > > There are other and better methods to increase efficiency, for > > > > > > > > > > example, more memory for the cache, or a redesign of the app to use less > > > > > > > > > > cache memory. > > > > > > > > > > > > > > > > > > > > Jรถrg > > > > > > > > > > Am 15.10.2013 16:01 schrieb "subin ks" [ksubins321@gmail.com](mailto:ksubins321@gmail.com): > > > > > > > > > > > > > > > > > > > > > Hi Boaz, > > > > > > > > > > > > > > > > > > > > > > > Remove the "index.fielddata.cache: soft" line. It only puts a > > > > > > > > > > > > load on the GC and doesn't allow you to control how much you want allocated > > > > > > > > > > > > to it. > > > > > > > > > > > > > > > > > > > > > > I decided to set it to soft owing to suggestions from the group > > > > > > > > > > > here[https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14](https://groups.google.com/forum/#!topic/elasticsearch/VYcQumkzk14). > > > > > > > > > > > As said, it would help to prevent outofmemory issues. Therefore I had set > > > > > > > > > > > it to soft. I'd like to know your opinion too ๐Ÿ™‚ > > > > > > > > > > > > > > > > > > > > > > > Was there any reason for you change these? The defaults are > > > > > > > > > > > > usually good to go with. > > > > > > > > > > > > index.translog.flush\_\*\*threshold\_ops: 2000 > > > > > > > > > > > > indices.memory.index\_buffer\_\*\*size: 40% > > > > > > > > > > > > > > > > > > > > > > I'll set the above to defaults and let you know. Does the > > > > > > > > > > > translog setting has anything to do with the number of documents handled by > > > > > > > > > > > ES per sec? Is it set according to that basis? > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > Re, > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > > > . > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > > . > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 16, 2013, 7:07pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/16 "2013-10-16T19:07:07Z")
Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK 6, it is known to have some odd memory behavior. Jรถrg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [October 22, 2013, 5:43am UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/17 "2013-10-22T05:43:47Z")
Hello Jorg, Thanks for the suggestion. Seeing memory improvements by using Oracle Java 7. But I withdrew 1 log-forwarding server from the cluster. I've avoided "index.fielddata.cache: soft", infact I removed it. I have a few doubts about how caches (index and field) works in ES. I'll raise a separate thread for it. Thanks, * * * Subin \* \* On Thu, Oct 17, 2013 at 12:37 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK > 6, it is known to have some odd memory behavior. > > Jรถrg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [November 11, 2013, 2:56pm UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/18 "2013-11-11T14:56:40Z")
I've increased my "indices.fielddata.cache.size" from 4 to 30%. No issues yet. So moving on to Oracle Java 7, disabling "index.fielddata.cache: soft" (removing it altogether) and increasing caching from 4 to 30% made improvements for me. I have also increased "index.refresh\_interval" to 120s. Earlier it was 10s. Thanks to Boaz and Jorg. * * * Subin On Tue, Oct 22, 2013 at 11:13 AM, subin ks [ksubins321@gmail.com](mailto:ksubins321@gmail.com) wrote: > Hello Jorg, > > Thanks for the suggestion. Seeing memory improvements by using Oracle Java > 7. But I withdrew 1 log-forwarding server from the cluster. I've avoided > "index.fielddata.cache: soft", infact I removed it. I have a few doubts > about how caches (index and field) works in ES. I'll raise a separate > thread for it. > > Thanks, > > * * * > > Subin > > On Thu, Oct 17, 2013 at 12:37 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > Try Oracle Java 7, your memory problems will be less. Do not use OpenJDK > > 6, it is known to have some odd memory behavior. > > > > Jรถrg > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/YwdtIj4rFTo/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/yet-another-memory-usage-issue-es-0-90-5/13922/19 "2017-07-06T02:07:50Z")