Breaking Change: _id is not longer indexed


(Shay Banon) #1

Heya,

In order to improve memory usage per shard, the _id field is not longer indexed by default. Note, no capability is lost except for the ability to use search queries using the _id field. For that, a new ids query / filter was introduced which does not require the _id field to be indexed (and is actually faster). There is a setting to bring it back if needed.

Issue: https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter: https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with almost all functionality retained (with a bit slower type filtering) and a new type filter, though this does not improve memory consumption by any noticeable measure.

-shay.banon


(Sebastian Gavarini) #2

Hi Shay,

I have seen there is also an indexed field in Lucene's indices called "uid",
concatenating "type#id", which is taking a lot of space because the ids are
already unique. Is this still going to be created with this change?

Thanks,
Sebastian.

On Tue, Apr 19, 2011 at 8:54 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Heya,

In order to improve memory usage per shard, the _id field is not longer
indexed by default. Note, no capability is lost except for the ability to
use search queries using the _id field. For that, a new ids query / filter
was introduced which does not require the _id field to be indexed (and is
actually faster). There is a setting to bring it back if needed.

Issue: https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter:
https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with
almost all functionality retained (with a bit slower type filtering) and a
new type filter, though this does not improve memory consumption by any
noticeable measure.

-shay.banon


(Shay Banon) #3

Ids are not required to be unique across types. The uid will always be created, the _id will not. You need to have a way to identify the doc.
On Wednesday, April 20, 2011 at 5:08 AM, Sebastian Gavarini wrote:

Hi Shay,

I have seen there is also an indexed field in Lucene's indices called "uid", concatenating "type#id", which is taking a lot of space because the ids are already unique. Is this still going to be created with this change?

Thanks,
Sebastian.

On Tue, Apr 19, 2011 at 8:54 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Heya,

In order to improve memory usage per shard, the _id field is not longer indexed by default. Note, no capability is lost except for the ability to use search queries using the _id field. For that, a new ids query / filter was introduced which does not require the _id field to be indexed (and is actually faster). There is a setting to bring it back if needed.

Issue: https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter: https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with almost all functionality retained (with a bit slower type filtering) and a new type filter, though this does not improve memory consumption by any noticeable measure.

-shay.banon


(Grant Rodgers) #4

Does this remove the ability to order by _id?

On Apr 19, 4:54 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

In order to improve memory usage per shard, the _id field is not longer indexed by default. Note, no capability is lost except for the ability to use search queries using the _id field. For that, a new ids query / filter was introduced which does not require the _id field to be indexed (and is actually faster). There is a setting to bring it back if needed.

Issue:https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter:https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with almost all functionality retained (with a bit slower type filtering) and a new type filter, though this does not improve memory consumption by any noticeable measure.

-shay.banon


(Shay Banon) #5

Yes.
On Sunday, April 24, 2011 at 12:58 AM, Grant Rodgers wrote:

Does this remove the ability to order by _id?

On Apr 19, 4:54 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

In order to improve memory usage per shard, the _id field is not longer indexed by default. Note, no capability is lost except for the ability to use search queries using the _id field. For that, a new ids query / filter was introduced which does not require the _id field to be indexed (and is actually faster). There is a setting to bring it back if needed.

Issue:https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter:https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with almost all functionality retained (with a bit slower type filtering) and a new type filter, though this does not improve memory consumption by any noticeable measure.

-shay.banon


(Shay Banon) #6

Though, to add on that, there is a setting to set to make sure it is still indexed when upgrading. And, if starting fresh, you can mark it as indexed. I plan to also add a special sort option for id, that will be less memory heavy.
On Sunday, April 24, 2011 at 9:35 AM, Shay Banon wrote:

Yes.
On Sunday, April 24, 2011 at 12:58 AM, Grant Rodgers wrote:

Does this remove the ability to order by _id?

On Apr 19, 4:54 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

In order to improve memory usage per shard, the _id field is not longer indexed by default. Note, no capability is lost except for the ability to use search queries using the _id field. For that, a new ids query / filter was introduced which does not require the _id field to be indexed (and is actually faster). There is a setting to bring it back if needed.

Issue:https://github.com/elasticsearch/elasticsearch/issues/868.
Ids Query/Filter:https://github.com/elasticsearch/elasticsearch/issues/865.

As a side note, the _type mapping can also be set to index : no, with almost all functionality retained (with a bit slower type filtering) and a new type filter, though this does not improve memory consumption by any noticeable measure.

-shay.banon


(system) #7