Fast Multiple Id Query


(davrob) #1

Hi,

I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:

   "results for index <indexname> where index id IN

(1,4,5,8,...... 100s or 1000s of ids)"

I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.

Best Regards,

David.


(Shay Banon) #2

You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.
On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:

Hi,

I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:

"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"

I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.

Best Regards,

David.


(Karussell) #3

why not do a terms query against the special "_id" field?
I'm using that one with success :slight_smile:

Regards,
Peter.

On 27 Feb., 20:29, Shay Banon shay.ba...@elasticsearch.com wrote:

You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.

On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:

Hi,

I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:

"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"

I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.

Best Regards,

David.


(Shay Banon) #4

Yes, thats certainly possible as well. A multi get API can be a bit more optimized than that (but not much, especially if you use query_and_fetch search type).
On Tuesday, March 1, 2011 at 10:05 PM, Karussell wrote:

why not do a terms query against the special "_id" field?
I'm using that one with success :slight_smile:

Regards,
Peter.

On 27 Feb., 20:29, Shay Banon shay.ba...@elasticsearch.com wrote:

You can use the async API to load the ids in parallel. There will be a bulk get API in the near future that will try and optimize that. Note, make sure there is enough mem both on the ES servers and on your client to handle 1000s of json representations.

On Sunday, February 27, 2011 at 2:50 PM, davrob2 wrote:

Hi,

I want to create a Join index that will hold the relationship between
two other indexes' ids. In order for this to work I will need to do a
query that does the following, very quickly using the Java API, once I
have a list of Ids that I want to turn into JSON:

"results for index where index id IN
(1,4,5,8,...... 100s or 1000s of ids)"

I could do a bulk query with hundreds of separate id queries or
iterate through each Id separately, but that would be very
inefficient. Any ideas to do this efficiently appreciated.

Best Regards,

David.


(system) #5