# Elasticsearch Memory issue **URL:** \ **Category:** Elasticsearch\ **Created:** [June 30, 2014, 3:04pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398 "2014-06-30T15:04:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![AlexK](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@AlexK](https://discuss.elastic.co/u/AlexK)\ **Post date:** [June 30, 2014, 3:04pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/1 "2014-06-30T15:04:11Z")
Hi, I am using ES version 1.1.1 on a _single node_ with the below settings: - 2 shards per indices; - currently 395 indices; - currently _428GB_; - _18GB_ Heap commited on a 24 GB machine; I used it only for indexing now, because I noted some memory issues when performing search: elasticsearch became unresponsive... My requirement curently is to use ElasticSearch only on a single machine configuration. I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge Collector very often) whithout success. Does anyone have some advices for me ? 1. Is the only way is to increase the heap (do you have a ratio RAM / data size) ? 2. Does adding additional nodes on the same machine will help, why ? 3. Final question, I cannot see why in the indexing part so much of the RAM is used, if anybody can explain ? Thanks a lot for your answers. Alex. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 30, 2014, 10:25pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/2 "2014-06-30T22:25:07Z")
You are more than likely reaching the limits of the node. Your options are to delete data, add more RAM (you should have 50% of system for heap), close some old indexes or add nodes. Adding more nodes spreads the shards of your indexes across the cluster which is essentially spreading the load. You could try disabling bloom filter cache as this will reduce your memory usage a little, take a look Elasticsearch Curator which can do this for you. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 1 July 2014 01:04, AlexK [alexandre.kapell@gmail.com](mailto:alexandre.kapell@gmail.com) wrote: > Hi, > > I am using ES version 1.1.1 on a _single node_ with the below settings: > > - 2 shards per indices; > - currently 395 indices; > - currently _428GB_; > - _18GB_ Heap commited on a 24 GB machine; > > I used it only for indexing now, because I noted some memory issues when > performing search: elasticsearch became unresponsive... > My requirement curently is to use Elasticsearch only on a single machine > configuration. > > I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge > Collector very often) whithout success. > > Does anyone have some advices for me ? > > 1. Is the only way is to increase the heap (do you have a ratio RAM / data > size) ? > 2. Does adding additional nodes on the same machine will help, why ? > 3. Final question, I cannot see why in the indexing part so much of the > RAM is used, if anybody can explain ? > > Thanks a lot for your answers. > Alex. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YSdjUQnQJ\_Wwu82iHxT5m%3DZVDRR\_pskCXQ2RPXmLHRxQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YSdjUQnQJ_Wwu82iHxT5m%3DZVDRR_pskCXQ2RPXmLHRxQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![AlexK](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@AlexK](https://discuss.elastic.co/u/AlexK)\ **Post date:** [July 1, 2014, 7:30am UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/3 "2014-07-01T07:30:16Z")
Hi Mark, Thanks for your quick answer. I cannot increase the RAM for ES, as I am already using 75% of the ram for the JVM. I will take a look at disabling the bloom filter cache to see if that change anything. Regarding the option of adding more nodes: - Do you have an idea of how many nodes are required to sustain that quantity of data (428Gb), I will say 3 nodes one for each indice and an extra node for replica, do you agree ? - In the hypothesis that I will not use in a first time replicas, do you have an idea about the quantity of RAM needed for the two nodes (continuous indexing and few search requests) ? Again, thanks for your answers. Alex. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 1, 2014, 7:35am UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/4 "2014-07-01T07:35:07Z")
When it comes to capacity the answer is, it depends. Given you're at around 430GB on a single node now, I'd add another node and then see how things look at around the 8-900GB mark (spread across both). Another clarification; The recommended operating procedure is to use half your system RAM for heap, leaving the other half for OS caching, which increases performance of ES. In your case under the best possible circumstances, you should only really be using 12GB of your total 24GB. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 1 July 2014 17:30, AlexK [alexandre.kapell@gmail.com](mailto:alexandre.kapell@gmail.com) wrote: > Hi Mark, > > Thanks for your quick answer. > I cannot increase the RAM for ES, as I am already using 75% of the ram for > the JVM. > I will take a look at disabling the bloom filter cache to see if that > change anything. > > Regarding the option of adding more nodes: > > - Do you have an idea of how many nodes are required to sustain that > quantity of data (428Gb), I will say 3 nodes one for each indice and an > extra node for replica, do you agree ? > - In the hypothesis that I will not use in a first time replicas, do you > have an idea about the quantity of RAM needed for the two nodes (continuous > indexing and few search requests) ? > > Again, thanks for your answers. > Alex. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Z5dt7GP%3DnhfijLBUc\_nuaswMQ6gzGgG0FuM4i4BfSMoA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Z5dt7GP%3DnhfijLBUc_nuaswMQ6gzGgG0FuM4i4BfSMoA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [July 1, 2014, 10:38am UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/5 "2014-07-01T10:38:54Z")
You should tweak cache sizes. At least the field data cache needs to be restricted (unbounded by default). Also, ensuring the various circuit breakers are turned on will help. Another tip is to disable the \_all field if you don't need it. All this should reduce the amount of memory ES uses and make it less likely your cluster becomes unavailable. We use Elasticsearch with Kibana in our production setup. Things definitely do not fail gracefully if you run short of memory; so you need to prevent that situation. I've had a completely unresponsive cluster on two occasions. With the current settings, it has been running stable for several weeks now. I've learned a few of these things the hard way. I think a ES tuning guide for non experts is desperately needed. The out of the box experience is not really appropriate for any serious production environment. But then, you wouldn't run mysql with default settings in production either. In my experience, you currently need to piece together bits of good advice spread through the documentation and various forum posts. If you have an untuned Elasticsearch in production there are several failure scenarios that are likely to result in unavailability and data loss. Especially if you are using ELK with lots of log data, you need to tune or you basically will have a dead cluster in no time due to OOMs. Jilles On Monday, June 30, 2014 5:04:12 PM UTC+2, AlexK wrote: > Hi, > > I am using ES version 1.1.1 on a _single node_ with the below settings: > > - 2 shards per indices; > - currently 395 indices; > - currently _428GB_; > - _18GB_ Heap commited on a 24 GB machine; > > I used it only for indexing now, because I noted some memory issues when > performing search: elasticsearch became unresponsive... > My requirement curently is to use Elasticsearch only on a single machine > configuration. > > I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge > Collector very often) whithout success. > > Does anyone have some advices for me ? > > 1. Is the only way is to increase the heap (do you have a ratio RAM / data > size) ? > 2. Does adding additional nodes on the same machine will help, why ? > 3. Final question, I cannot see why in the indexing part so much of the > RAM is used, if anybody can explain ? > > Thanks a lot for your answers. > Alex. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5e79bff1-409e-4389-8be3-69590f3ffb56%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5e79bff1-409e-4389-8be3-69590f3ffb56%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![AlexK](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@AlexK](https://discuss.elastic.co/u/AlexK)\ **Post date:** [July 1, 2014, 12:33pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/6 "2014-07-01T12:33:37Z")
Thanks both of you for the answer. I have just decrease the Java Jvm memory to 50% (12G), I will see if that helps. @Jilles: - I am using the defaut Logstash template and I things that by default the \_all field is disable...Ah no that not the case ☹ I will correct this settings but why by default this field is enabled ? - I will set a limit for the field cache. But regarding the circuit breaker, what are these ? - I totally agree that it would be good to have a deep tuning guide on ES 🙂 Thanks again, Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3894e4b5-eb8a-4c0d-90ff-75eb884743f9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3894e4b5-eb8a-4c0d-90ff-75eb884743f9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [July 1, 2014, 1:31pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/7 "2014-07-01T13:31:06Z")
My understanding was that ES 1.1 was using memory mapped files and so the field cache would not be part of the heap. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/308f68ba-bd7a-4775-ae1d-90088445d103%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/308f68ba-bd7a-4775-ae1d-90088445d103%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![AlexK](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@AlexK](https://discuss.elastic.co/u/AlexK)\ **Post date:** [July 1, 2014, 1:37pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/8 "2014-07-01T13:37:39Z")
Setting up the JVM memory to 50% (12G) did not ease the problem as I noticed GC collection up to 3min 🙂 Will really need to add a bunch of RAM to my machine.. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/874f127a-639a-4416-bdec-d50d77920f45%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/874f127a-639a-4416-bdec-d50d77920f45%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 1, 2014, 1:40pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/9 "2014-07-01T13:40:32Z")
With ES on a single machine, "tuning" does not cure the symptoms in the long run. ES was designed to scale out on many nodes, so the simplest path is to add nodes. In a restricted environment, you could try to disable features that consume a fairly amount of resources: disable \_source and \_all field, bloom codec filter cache, reduce complex analyzers in the mapping, reduce number of shards, reduce filter caches ... Jörg On Mon, Jun 30, 2014 at 5:04 PM, AlexK [alexandre.kapell@gmail.com](mailto:alexandre.kapell@gmail.com) wrote: > Hi, > > I am using ES version 1.1.1 on a _single node_ with the below settings: > > - 2 shards per indices; > - currently 395 indices; > - currently _428GB_; > - _18GB_ Heap commited on a 24 GB machine; > > I used it only for indexing now, because I noted some memory issues when > performing search: elasticsearch became unresponsive... > My requirement curently is to use Elasticsearch only on a single machine > configuration. > > I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge > Collector very often) whithout success. > > Does anyone have some advices for me ? > > 1. Is the only way is to increase the heap (do you have a ratio RAM / data > size) ? > 2. Does adding additional nodes on the same machine will help, why ? > 3. Final question, I cannot see why in the indexing part so much of the > RAM is used, if anybody can explain ? > > Thanks a lot for your answers. > Alex. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHB\_jCLt7%3DR4H4O\_wy-sAhGx90TrpW%3Du8AHXROdemNyYA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHB_jCLt7%3DR4H4O_wy-sAhGx90TrpW%3Du8AHXROdemNyYA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![AlexK](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@AlexK](https://discuss.elastic.co/u/AlexK)\ **Post date:** [July 1, 2014, 1:55pm UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/10 "2014-07-01T13:55:41Z")
Hi Jorg, Thanks for the answer. The idea behind the restriction of a single machine was for instance to install ELK on a machine and perform fast indexing and review of a set of log. What I got wrong is that the log size can be important (hundreds of Gb) so this architecture will not work, according to the answers above...(goal was to replace Splunk in a similar set up) Thanks, Alex. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/94ccef51-1cd9-4210-9f6c-d622108633df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94ccef51-1cd9-4210-9f6c-d622108633df%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:18am UTC](https://discuss.elastic.co/t/elasticsearch-memory-issue/18398/11 "2017-07-06T01:18:40Z")