Query Advice Needed

Elastic Stack Elasticsearch
1

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer

2

If those are small documents, then just getting all of them is fine. I plan to add a "no_sort" search type (or similar) to make it even less taxing. Make sure you use query_and_fetch, which will make it even faster (I assume you want all docs that match that query).
On Wednesday, May 4, 2011 at 12:13 AM, frazer wrote:

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer

3

Great, thanks Shay.

The documents have about 40 fields with really only 1 of those fields
potentially containing any sizeable text. Would that qualify as a
small document?

Also, I only really need ids, is there much of a performance gain
doing an id only query?

Yes, a no sort search type would be perfect for my scenario.

Thanks again

On May 3, 5:17 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

If those are small documents, then just getting all of them is fine. I plan to add a "no_sort" search type (or similar) to make it even less taxing. Make sure you use query_and_fetch, which will make it even faster (I assume you want all docs that match that query).

On Wednesday, May 4, 2011 at 12:13 AM, frazer wrote:

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer

4

If you just want to get the ids, then providing an empty fields array in the search request will return it, so the size of the document is not a problem. And, in this case, getting al the results back is certainly something that you should do.
On Wednesday, May 4, 2011 at 3:37 AM, frazer wrote:

Great, thanks Shay.

The documents have about 40 fields with really only 1 of those fields
potentially containing any sizeable text. Would that qualify as a
small document?

Also, I only really need ids, is there much of a performance gain
doing an id only query?

Yes, a no sort search type would be perfect for my scenario.

Thanks again

On May 3, 5:17 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

If those are small documents, then just getting all of them is fine. I plan to add a "no_sort" search type (or similar) to make it even less taxing. Make sure you use query_and_fetch, which will make it even faster (I assume you want all docs that match that query).

On Wednesday, May 4, 2011 at 12:13 AM, frazer wrote:

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer

5

Thanks Shay, so It sounds like its more efficient for me just to get
back the ids.

Thanks for the advice

On May 4, 2:44 am, Shay Banon shay.ba...@elasticsearch.com wrote:

If you just want to get the ids, then providing an empty fields array in the search request will return it, so the size of the document is not a problem. And, in this case, getting al the results back is certainly something that you should do.

On Wednesday, May 4, 2011 at 3:37 AM, frazer wrote:

Great, thanks Shay.

The documents have about 40 fields with really only 1 of those fields
potentially containing any sizeable text. Would that qualify as a
small document?

Also, I only really need ids, is there much of a performance gain
doing an id only query?

Yes, a no sort search type would be perfect for my scenario.

Thanks again

On May 3, 5:17 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

If those are small documents, then just getting all of them is fine. I plan to add a "no_sort" search type (or similar) to make it even less taxing. Make sure you use query_and_fetch, which will make it even faster (I assume you want all docs that match that query).

On Wednesday, May 4, 2011 at 12:13 AM, frazer wrote:

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer

6

Getting back just hte ids means much less data to get back. Its up to what you really need, getting the _source from ES compared to the DB will be faster (because its "already" there).
On Wednesday, May 4, 2011 at 2:50 PM, frazer wrote:

Thanks Shay, so It sounds like its more efficient for me just to get
back the ids.

Thanks for the advice

On May 4, 2:44 am, Shay Banon shay.ba...@elasticsearch.com wrote:

If you just want to get the ids, then providing an empty fields array in the search request will return it, so the size of the document is not a problem. And, in this case, getting al the results back is certainly something that you should do.

On Wednesday, May 4, 2011 at 3:37 AM, frazer wrote:

Great, thanks Shay.

The documents have about 40 fields with really only 1 of those fields
potentially containing any sizeable text. Would that qualify as a
small document?

Also, I only really need ids, is there much of a performance gain
doing an id only query?

Yes, a no sort search type would be perfect for my scenario.

Thanks again

On May 3, 5:17 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

If those are small documents, then just getting all of them is fine. I plan to add a "no_sort" search type (or similar) to make it even less taxing. Make sure you use query_and_fetch, which will make it even faster (I assume you want all docs that match that query).

On Wednesday, May 4, 2011 at 12:13 AM, frazer wrote:

I have a situation where I need to query an index and return a 1000
documents. This query will be frequently run by many users.

Am I best to

a) get the 1000 documents in 1 query
b) break the query down into smaller ones, manual paging
c) use the scan search type and scroll for 1k documents (sort is not
an issue)

Just wondering out of these options which would be most efficient.

Thanks in advance

Frazer