Performance hit when sorting on _id


(zohar) #1

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort to something else but was wondering if this was expected ?


(Shay Banon) #2

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.
On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?

View this message in context: http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-sorting-on-id-tp2026820p2026820.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(zohar) #3

All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?

View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Shay Banon) #4

All field values in the index, thats how sorting works...
On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:

All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?

View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(zohar) #5

Ouch. Good to know.
On Dec 6, 10:06 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

All field values in the index, thats how sorting works...

On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:

All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?

View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Shay Banon) #6

Nothing new for Lucene users..., though I have ideas on how to reduce the memory implications of it.
On Wednesday, December 8, 2010 at 12:01 AM, Zohar wrote:

Ouch. Good to know.
On Dec 6, 10:06 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

All field values in the index, thats how sorting works...

On Monday, December 6, 2010 at 11:45 PM, Zohar wrote:

All filed values in the index or all field values matching the query
or filter?

On Dec 6, 4:29 pm, Shay Banon wrote:

The first query when sorting on a field will cause all the field values to be loaded to memory. Since _id is unique, it has quite a few values, so it takes time to load them. but once its loaded, the next queries will be fast. Also, make sure you are not running into memory problems when doing it.

On Monday, December 6, 2010 at 1:51 PM, zohar wrote:

Hi
We are seeing a massive performance hit ( query goes from 200ms to 1min) for
a largish index if we sort on
_id. version is 0.11 we are upgrading as I type. We switch the default sort
to something else but was wondering if this was expected ?

View this message in context:http://elasticsearch-users.115913.n3.nabble.com/performance-hit-when-...
Sent from the ElasticSearch Users mailing list archive at Nabble.com.
.ba...@elasticsearch.com>


(system) #7