Omit_norms on _all field

dilshan · August 23, 2012, 1:25pm

Hi,

It seems like when I set "omit_norms : true" on a field, it affects the
results only when I search on that field directly. It doesn't work when I
search on _all field.
Also, after setting "omit_norms : true" on a field, it doesn't allow me to
change the setting back to false through the update mapping API.
Can somebody please explain me whether this is the intended behavior of
omit_norms and why it is working as that?
Please see below for the curl recreation of this.

gist.github.com

https://gist.github.com/dilshancalcey/3436362

gistfile1.txt

Create Index

curl -XPUT http://localhost:9200/tester/ -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "test" : {
            "properties" : {
                "name" : { "type" : "string", "index" : "analyzed", "omit_norms" : true, "include_in_all" : true }

This file has been truncated. show original

In the recreation, when searched on 'name' field it returns identical
scores. But when searched on '_all' field scores are different.
Also, 'omit_norms' value doesn't change after changing the mapping.

thanks,
Dilshan.

--

dilshan · August 26, 2012, 11:46am

Hi,

can somebody please help me with this?

thanks.

--

dilshan · October 10, 2012, 10:40pm

Any ideas on this?

thanks.

--

Clinton_Gormley · October 11, 2012, 7:43am

Hiya

It seems like when I set "omit_norms : true" on a field, it affects
the results only when I search on that field directly. It doesn't work
when I search on _all field.

Correct. The _all field is a completely separate field, with its own
settings and analysis process.

The values (not tokens!) from all fields are passed to the _all field,
which then analyses them using (by default) the standard analyser.

So what you are seeing is expected

clint

--

dilshan · October 18, 2012, 4:34am

Hi,

Thanks for the reply.
I guess I got why it doesn't work on _all field.
Any idea why we can't set the omit_norms field back to false through update
mapping API ?

thanks.

--

Clinton_Gormley · October 18, 2012, 7:38am

Hiya

Thanks for the reply.
I guess I got why it doesn't work on _all field.
Any idea why we can't set the omit_norms field back to false through
update mapping API ?

You can't make changes to existing fields with update mapping, only add
new fields (or new multi_fields), as your existing data would be
incorrect,

For that you need reindex your data.

But why not just query the original field(s) instead of the _all field?

clint

thanks.

--

--

dilshan · October 22, 2012, 6:33am

On Thursday, October 18, 2012 1:09:10 PM UTC+5:30, Clinton Gormley wrote:

Hiya

Thanks for the reply.
I guess I got why it doesn't work on _all field.
Any idea why we can't set the omit_norms field back to false through
update mapping API ?

You can't make changes to existing fields with update mapping, only add
new fields (or new multi_fields), as your existing data would be
incorrect,

For that you need reindex your data.

But why not just query the original field(s) instead of the _all field?

clint

There are about 20 boosted fields we are indexing and we observe that
querying on _all field gives us much better performance compared to
querying separately on each field.
Basically, what we are trying to achieve is to avoid the impact of document
size when calculating ES score.
Thought omit_norms is the solution here but it seems I cannot get it to
work, first because it doesn't work on _all field and second I cannot set
it in a boosted field.
Any idea on avoiding impact of document size when calculating ES score in
such a case?

Thanks.

--