# Fault tolerant elasticsearch (JVM Heap OOM) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2013, 4:03pm UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413 "2013-04-02T16:03:49Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![foufos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/foufos/32/2388_2.png) [@foufos](https://discuss.elastic.co/u/foufos)\ **Post date:** [April 2, 2013, 4:03pm UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/1 "2013-04-02T16:03:49Z")
Dear elasticsearch users I am running a PHP web application whose data layer is based out of 3 elasticsearch nodes. Once in a while there might be an individual node failing (e.g. recently one run into a JVM Heap OOM) but the cluster would still become green (2 nodes required) so I would like to make the application fault tollerant. What is the best practice to avoid sending requests to the instance that fails? Would you implement healthchecks at the application layer? Any examples or advice would be much appreciated -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 2, 2013, 4:07pm UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/2 "2013-04-02T16:07:55Z")
What are you exactly doing? Are you indexing documents? Are you searching for documents? Do you keep your queries in a logfile so you can trace what is going on? Did you enable logging at GC level in Elasticsearch? Have you a strategy for sizing your application, that is, have you calculated in advance how much resources you will need? Jörg Am 02.04.13 18:03, schrieb foufos: > Dear elasticsearch users > > I am running a PHP web application whose data layer is based out of 3 > elasticsearch nodes. > > Once in a while there might be an individual node failing (e.g. > recently one run into a JVM Heap OOM) but the cluster would still > become green (2 nodes required) so I would like to make the > application fault tollerant. > > What is the best practice to avoid sending requests to the instance > that fails? > > Would you implement healthchecks at the application layer? > > ## Any examples or advice would be much appreciated > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![q42jaap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/q42jaap/32/2241_2.png) [@q42jaap](https://discuss.elastic.co/u/q42jaap)\ **Post date:** [April 3, 2013, 4:41pm UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/3 "2013-04-03T16:41:18Z")
We've considered using haproxy to loadbalance (round-robin) the REST calls to the different nodes. haproxy can easily do a health check, by pinging the machine or sending HTTP requests and checking the response. In the end we ended up with an off-the-shelve-loadbalancer from our hosting company. This seems to work just fine, but we haven't tested that ourselves. Jaap Jaap Taal [Q42 BV | tel 070 44523 42 | direct 070 44523 65 | [http://q42.nl](http://q42.nl) | Waldorpstraat 17F, Den Haag | Vijzelstraat 72 unit 4.23, Amsterdam | KvK 30164662 ] On Tue, Apr 2, 2013 at 6:07 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > What are you exactly doing? Are you indexing documents? Are you searching > for documents? Do you keep your queries in a logfile so you can trace what > is going on? Did you enable logging at GC level in Elasticsearch? Have you > a strategy for sizing your application, that is, have you calculated in > advance how much resources you will need? > > Jörg > > Am 02.04.13 18:03, schrieb foufos: > > Dear elasticsearch users > > > I am running a PHP web application whose data layer is based out of 3 > > elasticsearch nodes. > > > > Once in a while there might be an individual node failing (e.g. recently > > one run into a JVM Heap OOM) but the cluster would still become green (2 > > nodes required) so I would like to make the application fault tollerant. > > > > What is the best practice to avoid sending requests to the instance that > > fails? > > > > Would you implement healthchecks at the application layer? > > > > ## Any examples or advice would be much appreciated > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 4, 2013, 6:33am UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/4 "2013-04-04T06:33:33Z")
Hey another solution might be to run a client node on your web application server. This is an elasticsearch node, which does not hold any data and is not allowed to become master, but still knows the clusters internal structure and which nodes can be queried (and a little bit more). There is a comment about that configuration in the default elasticsearch.yml configuration as well (which is not the best place to put it obvously). --Alex On Tue, Apr 2, 2013 at 6:03 PM, foufos [foufos7@gmail.com](mailto:foufos7@gmail.com) wrote: > Dear elasticsearch users > > I am running a PHP web application whose data layer is based out of 3 > elasticsearch nodes. > > Once in a while there might be an individual node failing (e.g. recently > one run into a JVM Heap OOM) but the cluster would still become green (2 > nodes required) so I would like to make the application fault tollerant. > > What is the best practice to avoid sending requests to the instance that > fails? > > Would you implement healthchecks at the application layer? > > Any examples or advice would be much appreciated > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 4, 2013, 9:37am UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/5 "2013-04-04T09:37:37Z")
@jorge What are the most common causes for OOM in ES? @foufos I know if you use the java client and all addresses to the transport client it will manage this for you. otherwise you can just have a list of servers to try your request against if you don't want a load balancer (I am assuming those would be the only ways if you opt for using the REST api via http) On Thursday, April 4, 2013 10:33:33 AM UTC+4, Alexander Reelsen wrote: > Hey > > another solution might be to run a client node on your web application > server. This is an elasticsearch node, which does not hold any data and is > not allowed to become master, but still knows the clusters internal > structure and which nodes can be queried (and a little bit more). There is > a comment about that configuration in the default elasticsearch.yml > configuration as well (which is not the best place to put it obvously). > > --Alex > > On Tue, Apr 2, 2013 at 6:03 PM, foufos \<[fou...@gmail.com](mailto:fou...@gmail.com) \\>wrote: > > > Dear elasticsearch users > > > > I am running a PHP web application whose data layer is based out of 3 > > elasticsearch nodes. > > > > Once in a while there might be an individual node failing (e.g. recently > > one run into a JVM Heap OOM) but the cluster would still become green (2 > > nodes required) so I would like to make the application fault tollerant. > > > > What is the best practice to avoid sending requests to the instance that > > fails? > > > > Would you implement healthchecks at the application layer? > > > > Any examples or advice would be much appreciated > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 4, 2013, 12:28pm UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/6 "2013-04-04T12:28:03Z")
OOM happens when the heap size is not sufficient. In ES it has to be considered for what workloads heap space is required: - for large segment merging. The bigger the index grows, the more heap is required for segment merging - for large documents and large bulks while indexing - for large result sets - and for field caching for filtering and faceting Finding a reasonable heap size requires some testing under different workloads. There is no general rule for a "correct" heap size. You can tackle OOM with scaling out (adding more nodes) or scaling up (add more RAM per node) or streamline the resource consumption during the lifecycle of the ES process (smaller segments while merging, smaller bulk indexing, smaller query results, avoiding "bad" queries with too heavy resource consumption) Jörg Am 04.04.2013 11:37, schrieb Mo: > @jorge What are the most common causes for OOM in ES? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![foufos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/foufos/32/2388_2.png) [@foufos](https://discuss.elastic.co/u/foufos)\ **Post date:** [April 12, 2013, 10:38am UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/7 "2013-04-12T10:38:28Z")
After investigation it turns out we had a lot of exceptions due to wrong mapping attributes. After fixing this, we haven't experienced another similar issue. can this cause OOM? @Jorg We have about 850K documents fairly small in size. and we also have routing set up to have less overhead. So the problem is temporarily not triggered but we have to create a fall back in another server becomes unresponsive and we get another split brain scenario So now we are considering of implementing a solution along the lines @alex suggested. You think that by doing something like that you can avoid a split brain scenario? thank you foufos On Thursday, 4 April 2013 15:28:03 UTC+3, Jörg Prante wrote: > OOM happens when the heap size is not sufficient. > > In ES it has to be considered for what workloads heap space is required: > > - for large segment merging. The bigger the index grows, the more heap > is required for segment merging > - for large documents and large bulks while indexing > - for large result sets > - and for field caching for filtering and faceting > > Finding a reasonable heap size requires some testing under different > workloads. There is no general rule for a "correct" heap size. > > You can tackle OOM with scaling out (adding more nodes) or scaling up > (add more RAM per node) or streamline the resource consumption during > the lifecycle of the ES process (smaller segments while merging, smaller > bulk indexing, smaller query results, avoiding "bad" queries with too > heavy resource consumption) > > Jörg > > Am 04.04.2013 11:37, schrieb Mo: > > > @jorge What are the most common causes for OOM in ES? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:41am UTC](https://discuss.elastic.co/t/fault-tolerant-elasticsearch-jvm-heap-oom/11413/8 "2017-07-06T02:41:22Z")