I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:
"results for index <indexname> where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"
I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.
You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.
On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:
Hi,
I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:
"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"
I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.
You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.
On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:
Hi,
I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:
"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"
I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.
Yes, thats certainly possible as well. A multi get API can be a bit more optimized than that (but not much, especially if you use query_and_fetch search type).
On Tuesday, March 1, 2011 at 10:05 PM, Karussell wrote:
why not do a terms query against the special "_id" field?
I'm using that one with success
You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.
On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:
Hi,
I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:
"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"
I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
