Hi ES team
I am facing issues indexing large documents (~ 35 MB). Is there any size limitation to the documents that we index? We are using nested type and nested query and it is working fine for smaller documents. But when I try a large document, the ES client hangs. Same issue when I tried using curl command.
Do you have any suggestions in indexing large documents? I read about splitting into parent/child documents.
If we go this route, can we query the parent using haschild filter and return fields from both the parent as well as matching children?
There are no explicit size limitations. As long as you have enough memory,
it should work. Moreover, 35MB doesn't seem to be excessively large. It's
not very clear from your question when ES client hangs (during indexing or
during searching) and what you mean by hanging. Does it take very long time
to execute or it never comes back no matter how long you wait? Assuming
that it hangs during searching, could you run hot_threadhttp://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-hot-threads/command while your long running query is hanging and post the output here?
Any chance you can provide a repro of this issue?
Igor
On Tuesday, July 23, 2013 2:03:03 AM UTC-4, george wrote:
Hi ES team
I am facing issues indexing large documents (~ 35 MB). Is there any
size
limitation to the documents that we index? We are using nested type and
nested query and it is working fine for smaller documents. But when I try
a
large document, the ES client hangs. Same issue when I tried using curl
command.
Do you have any suggestions in indexing large documents? I read about
splitting into parent/child documents.
If we go this route, can we query the parent using haschild filter and
return fields from both the parent as well as matching children?
Hi Igor
Thanks for your reply. Sorry for not being clear. I am facing issue while indexing the data. We have 6GB ES_HEAP_SIZE. I am using the Java client. I got Out of memory exception and another time I tried I got TimeoutException. Will try few more options and will update you if I am able to insert the document.
In our example, we have a type called "nested" and around 160K nested documnets. In case of such big documents, do you suggest us using parent/child relationship? Reading the docs, it seems like we can retrieve data either from parent or from the child , but not from both using has_child/has_parent filter. This means that if we need data from both parent and child doc, do we have to issue 2 queries - 1 query to fetch all parent documents that has child documents matching a particular filter. Then fire another query to fetch the matching child document which has the above parent. Is there any way to have a single query?
I need to know more about the structure of your documents, how often
different parts of you document change and what queries you run on your
documents in order to recommend one or another. But your understanding of
parent/child issue is correct, you will have to execute 2 queries in order
to get both parents and children, but you can combine them into single
multi-searchhttp://www.elasticsearch.org/guide/reference/api/multi-search/request.
Igor
On Wednesday, July 24, 2013 2:05:51 AM UTC-4, george wrote:
Hi Igor
Thanks for your reply. Sorry for not being clear. I am facing issue
while
indexing the data. We have 6GB ES_HEAP_SIZE. I am using the Java client. I
got Out of memory exception and another time I tried I got
TimeoutException.
Will try few more options and will update you if I am able to insert the
document.
In our example, we have a type called "nested" and around 160K nested
documnets. In case of such big documents, do you suggest us using
parent/child relationship? Reading the docs, it seems like we can retrieve
data either from parent or from the child , but not from both using
has_child/has_parent filter. This means that if we need data from both
parent and child doc, do we have to issue 2 queries - 1 query to fetch all
parent documents that has child documents matching a particular filter.
Then
fire another query to fetch the matching child document which has the
above
parent. Is there any way to have a single query?
Hi Igor
We inserted a document of size 10MB (Original document is around 45MB).
Indexing took around 13 secs Searching took around 800ms. We have some
mvel scripts running on the ES server as well.
I am attaching a sample doc that we insert in our ES node and the
corresponding query. This doc is small for the sake of clarity. In our big
docs the # of points will increase to almost 160K. Please let me know if
there are any issues in our query.
Can we use multisearch so that we get a parent AND only the child docs of
that parent matching a filter criteria. I thought multisearch queries have
to be independent of each other. We have the first query that matches the
parent which has atleast a child matching a criteria. Another query to
fetch the matched child document which has the parent from the previous
query. How can these be combined. Its a join based on parent id; Is it
possible in multisearch?
I need to know more about the structure of your documents, how often
different parts of you document change and what queries you run on your
documents in order to recommend one or another. But your understanding of
parent/child issue is correct, you will have to execute 2 queries in order
to get both parents and children, but you can combine them into single
multi-searchhttp://www.elasticsearch.org/guide/reference/api/multi-search/request.
Igor
On Wednesday, July 24, 2013 2:05:51 AM UTC-4, george wrote:
Hi Igor
Thanks for your reply. Sorry for not being clear. I am facing issue
while
indexing the data. We have 6GB ES_HEAP_SIZE. I am using the Java client.
I
got Out of memory exception and another time I tried I got
TimeoutException.
Will try few more options and will update you if I am able to insert the
document.
In our example, we have a type called "nested" and around 160K nested
documnets. In case of such big documents, do you suggest us using
parent/child relationship? Reading the docs, it seems like we can
retrieve
data either from parent or from the child , but not from both using
has_child/has_parent filter. This means that if we need data from both
parent and child doc, do we have to issue 2 queries - 1 query to fetch
all
parent documents that has child documents matching a particular filter.
Then
fire another query to fetch the matching child document which has the
above
parent. Is there any way to have a single query?
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email]http://user/SendEmail.jtp?type=node&node=4038613&i=0
.
I have the same problem. My docs have a nested structure and the nested
part can have more then 5K records. When i do a search on the elasticsearch
the response time is very slow when i have this large docs with nested
type.
Another interesting thing: when i do a sort operation on this kind of
document the response time is very slow too.
Anyone have an ideia on what is the best practices to use nested types in
elasticsearch?
In the moment we have a cluster with 4 machines and each machine has 4GB of
RAM. Our cluster start to slow down when search on the complex documents
begins.
On Thursday, July 25, 2013 5:02:27 AM UTC-3, george wrote:
Hi Igor
We inserted a document of size 10MB (Original document is around 45MB).
Indexing took around 13 secs Searching took around 800ms. We have some
mvel scripts running on the ES server as well.
I am attaching a sample doc that we insert in our ES node and the
corresponding query. This doc is small for the sake of clarity. In our big
docs the # of points will increase to almost 160K. Please let me know if
there are any issues in our query.
Can we use multisearch so that we get a parent AND only the child docs of
that parent matching a filter criteria. I thought multisearch queries have
to be independent of each other. We have the first query that matches the
parent which has atleast a child matching a criteria. Another query to
fetch the matched child document which has the parent from the previous
query. How can these be combined. Its a join based on parent id; Is it
possible in multisearch?
I need to know more about the structure of your documents, how often
different parts of you document change and what queries you run on your
documents in order to recommend one or another. But your understanding of
parent/child issue is correct, you will have to execute 2 queries in order
to get both parents and children, but you can combine them into single
multi-search http://www.elasticsearch.org/guide/reference/api/multi-search/ request.
Igor
On Wednesday, July 24, 2013 2:05:51 AM UTC-4, george wrote:
Hi Igor
Thanks for your reply. Sorry for not being clear. I am facing issue
while
indexing the data. We have 6GB ES_HEAP_SIZE. I am using the Java client.
I
got Out of memory exception and another time I tried I got
TimeoutException.
Will try few more options and will update you if I am able to insert the
document.
In our example, we have a type called "nested" and around 160K nested
documnets. In case of such big documents, do you suggest us using
parent/child relationship? Reading the docs, it seems like we can
retrieve
data either from parent or from the child , but not from both using
has_child/has_parent filter. This means that if we need data from both
parent and child doc, do we have to issue 2 queries - 1 query to fetch
all
parent documents that has child documents matching a particular filter.
Then
fire another query to fetch the matching child document which has the
above
parent. Is there any way to have a single query?
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email] http://user/SendEmail.jtp?type=node&node=4038613&i=0.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.