Parent / child use memory. Sorting as well.
You are using function_score as well. So it's not really a simple basic query here.
Let's say that you have 2 million of child docs. Let's say an ID is 1 byte.
If you need to load all IDs in memory, then you need to allocate 2 mb. If you have only 1 mb it can not fit right?
So it produces OOM. It won't take longer time. It won't fit. That's all.
So, you have many choices:
- add more RAM
- remove parent / child feature
- add more nodes (more shards I guess)
I'd go for the first option if the second does not fit to your use case.
My 0.05 cents
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
Le 28 novembre 2013 at 23:45:54, Marcelo Elias Del Valle (mvallebr@gmail.com) a écrit:
Hi everyone,
First of all, thanks a lot for the answer. Me and Matheus work together.
In this case, there is a single query running on this machine, and this single query make the node throw an OutOfMemoryException. I understand m1.large would be better (we are actually using m1.medium), but I really can't understand why elastic search is throwing out of memory for a single query, even if it uses a lot of memory. Shouldn't ES be able to handle it, although taking longer to provide the answer?
I currently have just 7Gb of data on each node. We want to have 90 Gb of data per node. Using -Xmx = 1Gb, 7Gb throwed the out of memory. There is no way of making it work with more data per node? My point is: I will have a lot of data, but really few queries.
Have any one had this same problem before? I was reading something about using soft cache instead of resilient:
Do you think the described solution could be a good fit for my problem?
Best regards,
Marcelo Valle.
Em quinta-feira, 28 de novembro de 2013 19h45min00s UTC-2, Mark Walkom escreveu:
ES will issue OOM errors as system resources are finite, it sounds like you have multiple queries running that are just filling all available RAM.
However there are no recommendations for memory sizing as it depends on what you are doing with your data. In your case you might want to look at doubling your system RAM to start with, and assign 50% of that to your heap (50% of available system RAM for ES is best practice).
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com
On 29 November 2013 08:29, Matheus Salvia mathe...@gmail.com wrote:
Hello Everyone.
First of All, I'd like to ask you for some help with my machine.
I Currently have 2 small EC2 instances as my elasticsearch nodes (they have about 3.5Gb of memory each). As of this moment, I have 2.5 million documents indexed, in a very complex parent/child structure, and I need to constantly run various types of queries, including, faceted queries, has_child queries, has_parent queries, and sometimes the queries (automatically generated) can become VERY big (the queries themselves, not necessarily the results), and some other times I run simple queries to get many results and store them somewhere else.
As of today, I started getting OutOfMemory errors in my second node (namely "robin"), just by running the simplest query possible (POST /users {size: 0}), does this make any sense? Wouldn't ES be able to handle every query I made, but getting slower and slower if I increased the complexity?
also, I noticed I wasn't allocating much memory to my JVM (about 1.2Gb).
What is the recomended memory size for me to allocate for each node?
Are my two EC2 small instances able to handle such described utilization?
Is there a Volume X Memory X number of instances table I can check?
Below I'm attatching one example query and one example stacktrace.
Query: http://pastebin.com/hjDxG0jt
StackTrace: http://pastebin.com/A82Q6i3k
Thanks in Advance,
Matheus Salvia
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOJV22YjDsNL3igj3XFeXEZ6uZ4duiojEYEBVPBbqKMhbMyO7A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6dcd70f4-3cac-40ba-b41f-ec492b6aa6d6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5297cd46.555c55b5.3e14%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.