fieldNorm & queryNorm in explain api

usha2626 · February 12, 2014, 5:32am

Hi,

I want to know what are fieldNorm and queryNorm in the output displayed by
the explain api..how are they calculated?

Thanks
Usha

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a009d8bd-6646-492e-8dc0-1af682a78286%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · February 12, 2014, 6:30am

The norms are calculated by Lucene's TFIDF algorithm:
http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html

The field norm is encoded into a single byte, so there is a loss of
precision.

--
Ivan

On Tue, Feb 11, 2014 at 9:32 PM, usha2626@gmail.com wrote:

Hi,

I want to know what are fieldNorm and queryNorm in the output displayed by
the explain api..how are they calculated?

Thanks
Usha

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a009d8bd-6646-492e-8dc0-1af682a78286%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAzhPuEZ9H%2BN0pnC-xmWvscxitDie4%3DSVtjus_B%2Bp8GDA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Navneet_Mathpal · February 12, 2014, 7:56am

Ivan
formula is queryNorm(q) = queryNorm(sumOfSquaredWeights)http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/DefaultSimilarity.html#queryNorm(float) =
1 –––––––––––––– sumOfSquaredWeights½

The sum of squared weights (of the query terms) is computed by the query
Weighthttp://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/Weight.htmlobject. For example, a
BooleanQueryhttp://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/BooleanQuery.htmlcomputes this value as:

sumOfSquaredWeightshttp://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/Weight.html#getValueForNormalization() =
q.getBoost()http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/Query.html#getBoost()
2 · ∑ ( idf(t)http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html#formula_idf ·
t.getBoost()http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html#formula_termBoost )
2
t in q

for my following search result how it is calculating:

{

took: 16
timed_out: false
_shards: {
- total: 5
- successful: 5
- failed: 0
  }
hits: {
- total: 2
- max_score: 0.2712221
- hits: [
  - {
    - _shard: 3
    - _node: k8BXmkARRsaaYlUJTRpIqQ
    - _index: phone
    - _type: iphone
    - _id: 2
    - _score: 0.2712221
    - _source: {
      - title: iphone update
      - description: this section shows how to update yopur iphone
        to work smoothly in your iphone
        }
    - _explanation: {
      - value: 0.27122214
      - description: max of:
      - details: [
        
        {
        
        value: 0.27122214
        
        description: sum of:
        
        details: [
        
        {
        
        value: 0.13561107
        
        description: weight(title:iphone in 0)
        [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.13561107
        
        description: score(doc=0,freq=1.0 =
        termFreq=1.0 ), product of:
        
        details: [
        
        {
        
        value: 0.70710677
        
        description: queryWeight, product
        of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 2.3043842
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.19178301
        
        description: fieldWeight in 0,
        product of:
        
        details: [
        
        {
        
        value: 1
        
        description: tf(freq=1.0),
        with freq of:
        
        details: [
        
        {
        
        value: 1
        
        description:
        termFreq=1.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.625
        
        description: fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        
        {
        
        value: 0.13561107
        
        description: weight(title:update in 0)
        [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.13561107
        
        description: score(doc=0,freq=1.0 =
        termFreq=1.0 ), product of:
        
        details: [
        
        {
        
        value: 0.70710677
        
        description: queryWeight, product
        of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 2.3043842
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.19178301
        
        description: fieldWeight in 0,
        product of:
        
        details: [
        
        {
        
        value: 1
        
        description: tf(freq=1.0),
        with freq of:
        
        details: [
        
        {
        
        value: 1
        
        description:
        termFreq=1.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.625
        
        description: fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        ]
        }
        
        {
        
        value: 0.16369703
        
        description: sum of:
        
        details: [
        
        {
        
        value: 0.0958915
        
        description: weight(description:iphone in 0)
        [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.0958915
        
        description: score(doc=0,freq=2.0 =
        termFreq=2.0 ), product of:
        
        details: [
        
        {
        
        value: 0.70710677
        
        description: queryWeight, product
        of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 2.3043842
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.13561106
        
        description: fieldWeight in 0,
        product of:
        
        details: [
        
        {
        
        value: 1.4142135
        
        description: tf(freq=2.0),
        with freq of:
        
        details: [
        
        {
        
        value: 2
        
        description:
        termFreq=2.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.3125
        
        description: fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        
        {
        
        value: 0.067805536
        
        description: weight(description:update in 0)
        [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.067805536
        
        description: score(doc=0,freq=1.0 =
        termFreq=1.0 ), product of:
        
        details: [
        
        {
        
        value: 0.70710677
        
        description: queryWeight, product
        of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 2.3043842
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.095891505
        
        description: fieldWeight in 0,
        product of:
        
        details: [
        
        {
        
        value: 1
        
        description: tf(freq=1.0),
        with freq of:
        
        details: [
        
        {
        
        value: 1
        
        description:
        termFreq=1.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.3125
        
        description: fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        ]
        }
        ]
        }
        }
  - {
    - _shard: 2
    - _node: k8BXmkARRsaaYlUJTRpIqQ
    - _index: phone
    - _type: iphone
    - _id: 1
    - _score: 0.04500804
    - _source: {
      - title: iphone
      - description: welcome to iphone world
        }
    - _explanation: {
      - value: 0.04500804
      - description: max of:
      - details: [
        
        {
        
        value: 0.04500804
        
        description: product of:
        
        details: [
        
        {
        
        value: 0.09001608
        
        description: sum of:
        
        details: [
        
        {
        
        value: 0.09001608
        
        description: weight(title:iphone in 0)
        [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.09001608
        
        description: score(doc=0,freq=1.0
        = termFreq=1.0 ), product of:
        
        details: [
        
        {
        
        value: 0.29335263
        
        description: queryWeight,
        product of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.9560043
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: fieldWeight in
        0, product of:
        
        details: [
        
        {
        
        value: 1
        
        description: tf(freq=1.0),
        with freq of:
        
        details: [
        
        {
        
        value: 1
        
        description:
        termFreq=1.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 1
        
        description:
        fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        ]
        }
        
        {
        
        value: 0.5
        
        description: coord(1/2)
        }
        ]
        }
        
        {
        
        value: 0.02250402
        
        description: product of:
        
        details: [
        
        {
        
        value: 0.04500804
        
        description: sum of:
        
        details: [
        
        {
        
        value: 0.04500804
        
        description: weight(description:iphone
        in 0) [PerFieldSimilarity], result of:
        
        details: [
        
        {
        
        value: 0.04500804
        
        description: score(doc=0,freq=1.0
        = termFreq=1.0 ), product of:
        
        details: [
        
        {
        
        value: 0.29335263
        
        description: queryWeight,
        product of:
        
        details: [
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.9560043
        
        description: queryNorm
        }
        ]
        }
        
        {
        
        value: 0.15342641
        
        description: fieldWeight in
        0, product of:
        
        details: [
        
        {
        
        value: 1
        
        description: tf(freq=1.0),
        with freq of:
        
        details: [
        
        {
        
        value: 1
        
        description:
        termFreq=1.0
        }
        ]
        }
        
        {
        
        value: 0.30685282
        
        description: idf(docFreq=1,
        maxDocs=1)
        }
        
        {
        
        value: 0.5
        
        description:
        fieldNorm(doc=0)
        }
        ]
        }
        ]
        }
        ]
        }
        ]
        }
        
        {
        
        value: 0.5
        
        description: coord(1/2)
        }
        ]
        }
        ]
        }
        }
        ]
        }

}

On Wednesday, 12 February 2014 12:00:03 UTC+5:30, Ivan Brusic wrote:

The norms are calculated by Lucene's TFIDF algorithm:
TFIDFSimilarity (Lucene 4.6.0 API)

The field norm is encoded into a single byte, so there is a loss of
precision.

--
Ivan

On Tue, Feb 11, 2014 at 9:32 PM, <usha...@gmail.com <javascript:>> wrote:

Hi,

I want to know what are fieldNorm and queryNorm in the output displayed
by the explain api..how are they calculated?

Thanks
Usha

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a009d8bd-6646-492e-8dc0-1af682a78286%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d86972e-6cca-488f-97d2-e9a1edef752f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.