# Help with OutOfMemory and some other memory concerns
**URL:** https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639
**Category:** Elasticsearch
**Created:** [November 28, 2013, 9:29pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639 "2013-11-28T21:29:21Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Matheus\_Salvia](https://discuss.elastic.co/u/Matheus_Salvia)
#### Post date: [November 28, 2013, 9:29pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/1 "2013-11-28T21:29:21Z")
Hello Everyone.
First of All, I'd like to ask you for some help with my machine.
I Currently have 2 small EC2 instances as my elasticsearch nodes (they have
about 3.5Gb of memory each). As of this moment, I have 2.5 million
documents indexed, in a very complex parent/child structure, and I need to
constantly run various types of queries, including, faceted queries,
has\_child queries, has\_parent queries, and sometimes the queries
(automatically generated) can become VERY big (the queries themselves, not
necessarily the results), and some other times I run simple queries to get
many results and store them somewhere else.
As of today, I started getting OutOfMemory errors in my second node (namely
"robin"), just by running the simplest query possible (POST /users {size:
0}), does this make any sense? Wouldn't ES be able to handle _every_ query
I made, but getting slower and slower if I increased the complexity?
also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
What is the recomended memory size for me to allocate for each node?
Are my two EC2 small instances able to handle such described utilization?
Is there a Volume X Memory X number of instances table I can check?
Below I'm attatching one example query and one example stacktrace.
Query: [http://pastebin.com/hjDxG0jt](http://pastebin.com/hjDxG0jt)
StackTrace: [http://pastebin.com/A82Q6i3k](http://pastebin.com/A82Q6i3k)
Thanks in Advance,
Matheus Salvia
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 28, 2013, 9:42pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/2 "2013-11-28T21:42:37Z")
Some comments:
- For production we recommend to start with m1.large
- For production you should start with 3 nodes and set minimum master node to 2 to limit split brain issue
- About memory sizing, it depends also on your document size. Elasticsearch needs memory. If you use parent/child, elasticsearch needs to load ids of documents in memory to perform fast lookup.
- Memory size for HEAP recommended: 1/2 of the available RAM. The other half will used by the FileSystem cache.
My 0.05 cents
HTH
--
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
@dadoonet | @elasticsearchfr
Le 28 novembre 2013 at 22:29:24, Matheus Salvia ([matheus2740@gmail.com](mailto:matheus2740@gmail.com)) a écrit:
Hello Everyone.
First of All, I'd like to ask you for some help with my machine.
I Currently have 2 small EC2 instances as my elasticsearch nodes (they have about 3.5Gb of memory each). As of this moment, I have 2.5 million documents indexed, in a very complex parent/child structure, and I need to constantly run various types of queries, including, faceted queries, has\_child queries, has\_parent queries, and sometimes the queries (automatically generated) can become VERY big (the queries themselves, not necessarily the results), and some other times I run simple queries to get many results and store them somewhere else.
As of today, I started getting OutOfMemory errors in my second node (namely "robin"), just by running the simplest query possible (POST /users {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_ query I made, but getting slower and slower if I increased the complexity?
also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
What is the recomended memory size for me to allocate for each node?
Are my two EC2 small instances able to handle such described utilization?
Is there a Volume X Memory X number of instances table I can check?
Below I'm attatching one example query and one example stacktrace.
Query: [http://pastebin.com/hjDxG0jt](http://pastebin.com/hjDxG0jt)
StackTrace: [http://pastebin.com/A82Q6i3k](http://pastebin.com/A82Q6i3k)
## Thanks in Advance, Matheus Salvia
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5297b8cd.6aa78f7f.3e14%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5297b8cd.6aa78f7f.3e14%40MacBook-Air-de-David.local).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 28, 2013, 9:45pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/3 "2013-11-28T21:45:00Z")
ES will issue OOM errors as system resources are finite, it sounds like you
have multiple queries running that are just filling all available RAM.
However there are no recommendations for memory sizing as it depends on
what you are doing with your data. In your case you might want to look at
doubling your system RAM to start with, and assign 50% of that to your heap
(50% of available system RAM for ES is best practice).
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 29 November 2013 08:29, Matheus Salvia [matheus2740@gmail.com](mailto:matheus2740@gmail.com) wrote:
> Hello Everyone.
> First of All, I'd like to ask you for some help with my machine.
> I Currently have 2 small EC2 instances as my elasticsearch nodes (they
> have about 3.5Gb of memory each). As of this moment, I have 2.5 million
> documents indexed, in a very complex parent/child structure, and I need to
> constantly run various types of queries, including, faceted queries,
> has\_child queries, has\_parent queries, and sometimes the queries
> (automatically generated) can become VERY big (the queries themselves, not
> necessarily the results), and some other times I run simple queries to get
> many results and store them somewhere else.
> As of today, I started getting OutOfMemory errors in my second node
> (namely "robin"), just by running the simplest query possible (POST /users
> {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_
> query I made, but getting slower and slower if I increased the complexity?
> also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
> What is the recomended memory size for me to allocate for each node?
> Are my two EC2 small instances able to handle such described utilization?
> Is there a Volume X Memory X number of instances table I can check?
>
> Below I'm attatching one example query and one example stacktrace.
>
> Query: [Example Query - Pastebin.com](http://pastebin.com/hjDxG0jt)
> StackTrace: [Caused by: org.elasticsearch.ElasticSearchException: java.lang.OutOfMemoryError: - Pastebin.com](http://pastebin.com/A82Q6i3k)
>
> Thanks in Advance,
> Matheus Salvia
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YBcbSmEqQCoey%3D4\_RYt4u\_1Q3Chd6\_016S2LMhLCr-Bw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YBcbSmEqQCoey%3D4_RYt4u_1Q3Chd6_016S2LMhLCr-Bw%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Marcelo\_Elias\_Del\_Va](https://discuss.elastic.co/u/Marcelo_Elias_Del_Va)
#### Post date: [November 28, 2013, 10:45pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/4 "2013-11-28T22:45:51Z")
Hi everyone,
First of all, thanks a lot for the answer. Me and Matheus work together.
In this case, there is a single query running on this machine, and this
single query make the node throw an OutOfMemoryException. I understand
m1.large would be better (we are actually using m1.medium), but I really
can't understand why Elasticsearch is throwing out of memory for a single
query, even if it uses a lot of memory. Shouldn't ES be able to handle it,
although taking longer to provide the answer?
I currently have just 7Gb of data on each node. We want to have 90 Gb of
data per node. Using -Xmx = 1Gb, 7Gb throwed the out of memory. There is no
way of making it work with more data per node? My point is: I will have a
lot of data, but really few queries.
Have any one had this same problem before? I was reading something about
using soft cache instead of resilient:
> **[ElasticSearch Cache Usage - Sematext](https://sematext.com/blog/elasticsearch-cache-usage/)**
>
> We’ve been doing a ton of work with ElasticSearch. Not long ago, we had a few situations where ElasticSearch would “eat” all the JVM heap memory we give it. It was so hungry, we could not feed it enough memory to keep it happy....
Do you think the described solution could be a good fit for my problem?
Best regards,
Marcelo Valle.
Em quinta-feira, 28 de novembro de 2013 19h45min00s UTC-2, Mark Walkom
escreveu:
> ES will issue OOM errors as system resources are finite, it sounds like
> you have multiple queries running that are just filling all available RAM.
>
> However there are no recommendations for memory sizing as it depends on
> what you are doing with your data. In your case you might want to look at
> doubling your system RAM to start with, and assign 50% of that to your heap
> (50% of available system RAM for ES is best practice).
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 29 November 2013 08:29, Matheus Salvia \<[mathe...@gmail.com](mailto:mathe...@gmail.com)\
>
> > wrote:
>
> > Hello Everyone.
> > First of All, I'd like to ask you for some help with my machine.
> > I Currently have 2 small EC2 instances as my elasticsearch nodes (they
> > have about 3.5Gb of memory each). As of this moment, I have 2.5 million
> > documents indexed, in a very complex parent/child structure, and I need to
> > constantly run various types of queries, including, faceted queries,
> > has\_child queries, has\_parent queries, and sometimes the queries
> > (automatically generated) can become VERY big (the queries themselves, not
> > necessarily the results), and some other times I run simple queries to get
> > many results and store them somewhere else.
> > As of today, I started getting OutOfMemory errors in my second node
> > (namely "robin"), just by running the simplest query possible (POST /users
> > {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_
> > query I made, but getting slower and slower if I increased the complexity?
> > also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
> > What is the recomended memory size for me to allocate for each node?
> > Are my two EC2 small instances able to handle such described utilization?
> > Is there a Volume X Memory X number of instances table I can check?
> >
> > Below I'm attatching one example query and one example stacktrace.
> >
> > Query: [Example Query - Pastebin.com](http://pastebin.com/hjDxG0jt)
> > StackTrace: [Caused by: org.elasticsearch.ElasticSearchException: java.lang.OutOfMemoryError: - Pastebin.com](http://pastebin.com/A82Q6i3k)
> >
> > Thanks in Advance,
> > Matheus Salvia
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 28, 2013, 11:09pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/5 "2013-11-28T23:09:58Z")
Parent / child use memory. Sorting as well.
You are using function\_score as well. So it's not really a simple basic query here.
Let's say that you have 2 million of child docs. Let's say an ID is 1 byte.
If you need to load all IDs in memory, then you need to allocate 2 mb. If you have only 1 mb it can not fit right?
So it produces OOM. It won't take longer time. It won't fit. That's all.
So, you have many choices:
- add more RAM
- remove parent / child feature
- add more nodes (more shards I guess)
I'd go for the first option if the second does not fit to your use case.
My 0.05 cents
--
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
@dadoonet | @elasticsearchfr
Le 28 novembre 2013 at 23:45:54, Marcelo Elias Del Valle ([mvallebr@gmail.com](mailto:mvallebr@gmail.com)) a écrit:
Hi everyone,
First of all, thanks a lot for the answer. Me and Matheus work together.
In this case, there is a single query running on this machine, and this single query make the node throw an OutOfMemoryException. I understand m1.large would be better (we are actually using m1.medium), but I really can't understand why elastic search is throwing out of memory for a single query, even if it uses a lot of memory. Shouldn't ES be able to handle it, although taking longer to provide the answer?
I currently have just 7Gb of data on each node. We want to have 90 Gb of data per node. Using -Xmx = 1Gb, 7Gb throwed the out of memory. There is no way of making it work with more data per node? My point is: I will have a lot of data, but really few queries.
Have any one had this same problem before? I was reading something about using soft cache instead of resilient:
> **[ElasticSearch Cache Usage](https://sematext.com/blog/elasticsearch-cache-usage/)**
>
> We’ve been doing a ton of work with ElasticSearch. Not long ago, we had a few situations where ElasticSearch would “eat” all the JVM heap memory we give it. It was so hungry, we could not feed it…
Do you think the described solution could be a good fit for my problem?
Best regards,
Marcelo Valle.
Em quinta-feira, 28 de novembro de 2013 19h45min00s UTC-2, Mark Walkom escreveu:
ES will issue OOM errors as system resources are finite, it sounds like you have multiple queries running that are just filling all available RAM.
However there are no recommendations for memory sizing as it depends on what you are doing with your data. In your case you might want to look at doubling your system RAM to start with, and assign 50% of that to your heap (50% of available system RAM for ES is best practice).
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 29 November 2013 08:29, Matheus Salvia [mathe...@gmail.com](mailto:mathe...@gmail.com) wrote:
Hello Everyone.
First of All, I'd like to ask you for some help with my machine.
I Currently have 2 small EC2 instances as my elasticsearch nodes (they have about 3.5Gb of memory each). As of this moment, I have 2.5 million documents indexed, in a very complex parent/child structure, and I need to constantly run various types of queries, including, faceted queries, has\_child queries, has\_parent queries, and sometimes the queries (automatically generated) can become VERY big (the queries themselves, not necessarily the results), and some other times I run simple queries to get many results and store them somewhere else.
As of today, I started getting OutOfMemory errors in my second node (namely "robin"), just by running the simplest query possible (POST /users {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_ query I made, but getting slower and slower if I increased the complexity?
also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
What is the recomended memory size for me to allocate for each node?
Are my two EC2 small instances able to handle such described utilization?
Is there a Volume X Memory X number of instances table I can check?
Below I'm attatching one example query and one example stacktrace.
Query: [http://pastebin.com/hjDxG0jt](http://pastebin.com/hjDxG0jt)
StackTrace: [http://pastebin.com/A82Q6i3k](http://pastebin.com/A82Q6i3k)
## Thanks in Advance, Matheus Salvia
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5297cd46.555c55b5.3e14%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5297cd46.555c55b5.3e14%40MacBook-Air-de-David.local).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Marcelo\_Elias\_Del\_Va](https://discuss.elastic.co/u/Marcelo_Elias_Del_Va)
#### Post date: [November 28, 2013, 11:13pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/6 "2013-11-28T23:13:01Z")
David,
```
Your answer helped much more than 5 cents, thanks. ;-)
```
Best regards,
Marcelo.
2013/11/28 David Pilato [david@pilato.fr](mailto:david@pilato.fr)
> Parent / child use memory. Sorting as well.
> You are using function\_score as well. So it's not really a simple basic
> query here.
>
> Let's say that you have 2 million of child docs. Let's say an ID is 1 byte.
> If you need to load all IDs in memory, then you need to allocate 2 mb. If
> you have only 1 mb it can not fit right?
> So it produces OOM. It won't take longer time. It won't fit. That's all.
>
> So, you have many choices:
>
> - add more RAM
> - remove parent / child feature
> - add more nodes (more shards I guess)
>
> I'd go for the first option if the second does not fit to your use case.
>
> My 0.05 cents
>
> --
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
>
> Le 28 novembre 2013 at 23:45:54, Marcelo Elias Del Valle (
> [mvallebr@gmail.com](mailto:mvallebr@gmail.com) //mvallebr@gmail.com) a écrit:
>
> Hi everyone,
>
> First of all, thanks a lot for the answer. Me and Matheus work together.
>
> In this case, there is a single query running on this machine, and this
> single query make the node throw an OutOfMemoryException. I understand
> m1.large would be better (we are actually using m1.medium), but I really
> can't understand why Elasticsearch is throwing out of memory for a single
> query, even if it uses a lot of memory. Shouldn't ES be able to handle it,
> although taking longer to provide the answer?
>
> I currently have just 7Gb of data on each node. We want to have 90 Gb of
> data per node. Using -Xmx = 1Gb, 7Gb throwed the out of memory. There is no
> way of making it work with more data per node? My point is: I will have a
> lot of data, but really few queries.
>
> Have any one had this same problem before? I was reading something about
> using soft cache instead of resilient:
> [ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/)
>
> Do you think the described solution could be a good fit for my problem?
>
> Best regards,
> Marcelo Valle.
>
> Em quinta-feira, 28 de novembro de 2013 19h45min00s UTC-2, Mark Walkom
> escreveu:
>
> > ES will issue OOM errors as system resources are finite, it sounds like
> > you have multiple queries running that are just filling all available RAM.
> >
> > However there are no recommendations for memory sizing as it depends on
> > what you are doing with your data. In your case you might want to look at
> > doubling your system RAM to start with, and assign 50% of that to your heap
> > (50% of available system RAM for ES is best practice).
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 29 November 2013 08:29, Matheus Salvia [mathe...@gmail.com](mailto:mathe...@gmail.com) wrote:
> >
> > > Hello Everyone.
> > > First of All, I'd like to ask you for some help with my machine.
> > > I Currently have 2 small EC2 instances as my elasticsearch nodes (they
> > > have about 3.5Gb of memory each). As of this moment, I have 2.5 million
> > > documents indexed, in a very complex parent/child structure, and I need to
> > > constantly run various types of queries, including, faceted queries,
> > > has\_child queries, has\_parent queries, and sometimes the queries
> > > (automatically generated) can become VERY big (the queries themselves, not
> > > necessarily the results), and some other times I run simple queries to get
> > > many results and store them somewhere else.
> > > As of today, I started getting OutOfMemory errors in my second node
> > > (namely "robin"), just by running the simplest query possible (POST /users
> > > {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_
> > > query I made, but getting slower and slower if I increased the complexity?
> > > also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
> > > What is the recomended memory size for me to allocate for each node?
> > > Are my two EC2 small instances able to handle such described utilization?
> > > Is there a Volume X Memory X number of instances table I can check?
> > >
> > > Below I'm attatching one example query and one example stacktrace.
> > >
> > > Query: [Example Query - Pastebin.com](http://pastebin.com/hjDxG0jt)
> > > StackTrace: [Caused by: org.elasticsearch.ElasticSearchException: java.lang.OutOfMemoryError: - Pastebin.com](http://pastebin.com/A82Q6i3k)
> > >
> > > ## Thanks in Advance, Matheus Salvia
> > >
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4du
> > > iojEYEBVPBbqKMhbMyO7A%[40mail.gmail.com](http://40mail.gmail.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
>
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/G52\_0euXegc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/G52_0euXegc/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/etPan.5297cd46.555c55b5.3e14%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5297cd46.555c55b5.3e14%40MacBook-Air-de-David.local)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
Marcelo Elias Del Valle
[http://mvalle.com](http://mvalle.com) - @mvallebr
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABKQidtarbzkjZ4idof40wKZmpF3A\_%3DSJ6gs9BsricCevCx9hA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABKQidtarbzkjZ4idof40wKZmpF3A_%3DSJ6gs9BsricCevCx9hA%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 28, 2013, 11:13pm UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/7 "2013-11-28T23:13:33Z")
When you run a query it needs to pull all the IDs for the associated
indexes into memory, coupled with the caching that ES does as part of
normal operations OOM with such a small amount of RAM as you are operating
on is to be expected.
Complex document structure will also impact this.
You can try adjusting your various caches, however that might only get you
so far.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 29 November 2013 09:45, Marcelo Elias Del Valle [mvallebr@gmail.com](mailto:mvallebr@gmail.com)wrote:
> Hi everyone,
>
> First of all, thanks a lot for the answer. Me and Matheus work together.
>
> In this case, there is a single query running on this machine, and this
> single query make the node throw an OutOfMemoryException. I understand
> m1.large would be better (we are actually using m1.medium), but I really
> can't understand why Elasticsearch is throwing out of memory for a single
> query, even if it uses a lot of memory. Shouldn't ES be able to handle it,
> although taking longer to provide the answer?
>
> I currently have just 7Gb of data on each node. We want to have 90 Gb of
> data per node. Using -Xmx = 1Gb, 7Gb throwed the out of memory. There is no
> way of making it work with more data per node? My point is: I will have a
> lot of data, but really few queries.
>
> Have any one had this same problem before? I was reading something about
> using soft cache instead of resilient:
> [ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/)
>
> Do you think the described solution could be a good fit for my problem?
>
> Best regards,
> Marcelo Valle.
>
> Em quinta-feira, 28 de novembro de 2013 19h45min00s UTC-2, Mark Walkom
> escreveu:
>
> > ES will issue OOM errors as system resources are finite, it sounds like
> > you have multiple queries running that are just filling all available RAM.
> >
> > However there are no recommendations for memory sizing as it depends on
> > what you are doing with your data. In your case you might want to look at
> > doubling your system RAM to start with, and assign 50% of that to your heap
> > (50% of available system RAM for ES is best practice).
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 29 November 2013 08:29, Matheus Salvia [mathe...@gmail.com](mailto:mathe...@gmail.com) wrote:
> >
> > > Hello Everyone.
> > > First of All, I'd like to ask you for some help with my machine.
> > > I Currently have 2 small EC2 instances as my elasticsearch nodes (they
> > > have about 3.5Gb of memory each). As of this moment, I have 2.5 million
> > > documents indexed, in a very complex parent/child structure, and I need to
> > > constantly run various types of queries, including, faceted queries,
> > > has\_child queries, has\_parent queries, and sometimes the queries
> > > (automatically generated) can become VERY big (the queries themselves, not
> > > necessarily the results), and some other times I run simple queries to get
> > > many results and store them somewhere else.
> > > As of today, I started getting OutOfMemory errors in my second node
> > > (namely "robin"), just by running the simplest query possible (POST /users
> > > {size: 0}), does this make any sense? Wouldn't ES be able to handle _every_
> > > query I made, but getting slower and slower if I increased the complexity?
> > > also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
> > > What is the recomended memory size for me to allocate for each node?
> > > Are my two EC2 small instances able to handle such described utilization?
> > > Is there a Volume X Memory X number of instances table I can check?
> > >
> > > Below I'm attatching one example query and one example stacktrace.
> > >
> > > Query: [Example Query - Pastebin.com](http://pastebin.com/hjDxG0jt)
> > > StackTrace: [Caused by: org.elasticsearch.ElasticSearchException: java.lang.OutOfMemoryError: - Pastebin.com](http://pastebin.com/A82Q6i3k)
> > >
> > > Thanks in Advance,
> > > Matheus Salvia
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4du
> > > iojEYEBVPBbqKMhbMyO7A%[40mail.gmail.com](http://40mail.gmail.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624au4vq6uA7tpAnVqhGVim%2ByqX60onPJ7thAezpUK7YxNQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624au4vq6uA7tpAnVqhGVim%2ByqX60onPJ7thAezpUK7YxNQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:04am UTC](https://discuss.elastic.co/t/help-with-outofmemory-and-some-other-memory-concerns/14639/8 "2017-07-06T02:04:16Z")