# Computing idf in elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [February 11, 2014, 2:40pm UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731 "2014-02-11T14:40:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sunayana\_choudhary](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sunayana\_choudhary](https://discuss.elastic.co/u/sunayana_choudhary)\ **Post date:** [February 11, 2014, 2:40pm UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/1 "2014-02-11T14:40:52Z")
Hi all, I have been analysing Elasticsearch results with explain:true condition, I am not able to understand what technique has been applied to calculate idf. I went through the lucene scoring formula i.e. idf(t) = 1+log(NumDocs/Doc frequency+1) Does not matches my results. Following is explanation for one of the results returned. \_explanation: { value: 5.8878393 description: weight(city:chicago in 1) [PerFieldSimilarity], result of: details: [ { value: 5.8878393 description: score(doc=1,freq=1.0 = termFreq=1.0 ), product of: details: [ { value: 0.99999994 description: queryWeight, product of: details: [ { value: 5.88784 description: idf(docFreq=2, maxDocs=398) } { value: 0.16984157 description: queryNorm } ] } { value: 5.88784 description: fieldWeight in 1, product of: details: [ { value: 1 description: tf(freq=1.0), with freq of: details: [ { value: 1 description: termFreq=1.0 } ] } { value: 5.88784 description: idf(docFreq=2, maxDocs=398) } { value: 1 description: fieldNorm(doc=1) } ] } ] } ] } } Thanks in advance.. 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c19fe1ea-aba7-4296-9536-cffc25a26836%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c19fe1ea-aba7-4296-9536-cffc25a26836%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 11, 2014, 3:12pm UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/2 "2014-02-11T15:12:46Z")
TF and IDF are calculated by shard, not per index, so the aggregated explanation might not have the exact numbers. Try changing your search type to a distributed one for more accurate results: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. How many shards does your index have? -- Ivan On Tue, Feb 11, 2014 at 6:40 AM, sunayana choudhary [sunayanacool@gmail.com](mailto:sunayanacool@gmail.com)wrote: > Hi all, > > I have been analysing Elasticsearch results with explain:true condition, I > am not able to understand what technique has been applied to calculate > idf. I went through the lucene scoring formula i.e. > > idf(t) = 1+log(NumDocs/Doc frequency+1) > > Does not matches my results. > Following is explanation for one of the results returned. > \_explanation: { > value: 5.8878393 > description: weight(city:chicago in 1) [PerFieldSimilarity], > result of: > details: [ > { > value: 5.8878393 > description: score(doc=1,freq=1.0 = termFreq=1.0 ), > product of: > details: [ > { > value: 0.99999994 > description: queryWeight, product of: > details: [ > { > value: 5.88784 > description: idf(docFreq=2, maxDocs=398) > } > { > value: 0.16984157 > description: queryNorm > } > ] > } > { > value: 5.88784 > description: fieldWeight in 1, product of: > details: [ > { > value: 1 > description: tf(freq=1.0), with freq of: > details: [ > { > value: 1 > description: termFreq=1.0 > } > ] > } > { > value: 5.88784 > description: idf(docFreq=2, maxDocs=398) > } > { > value: 1 > description: fieldNorm(doc=1) > } > ] > } > ] > } > ] > } > > } > > Thanks in advance.. 🙂 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c19fe1ea-aba7-4296-9536-cffc25a26836%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c19fe1ea-aba7-4296-9536-cffc25a26836%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAcGFL900aM1w2rodE3OL-jFD3YEsQw%2B3nEb0ZvswE2YQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAcGFL900aM1w2rodE3OL-jFD3YEsQw%2B3nEb0ZvswE2YQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 11, 2014, 6:22pm UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/3 "2014-02-11T18:22:47Z")
Also be aware that the log should be a natural log, i.e. the base is e instead of 10. So for example, pulling the first IDF from your results: value: 5.88784 description: idf(docFreq=2, maxDocs=398) idf = 1 + ln(398 / (2 + 1)) = 5.8878397166163280134321081764042 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/127b4c5c-09d7-4536-a587-4132863db3aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/127b4c5c-09d7-4536-a587-4132863db3aa%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 11, 2014, 8:27pm UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/4 "2014-02-11T20:27:55Z")
Oops, obvious answer. 🙂 I see questions about incorrect TFIDF scores and my mind automatically goes to DFS scoring (which is actually about TF, not IDF). -- Ivan On Tue, Feb 11, 2014 at 10:22 AM, Binh Ly [binh@hibalo.com](mailto:binh@hibalo.com) wrote: > Also be aware that the log should be a natural log, i.e. the base is e > instead of 10. So for example, pulling the first IDF from your results: > > value: 5.88784 > description: idf(docFreq=2, maxDocs=398) > idf = 1 + ln(398 / (2 + 1)) = 5.8878397166163280134321081764042 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/127b4c5c-09d7-4536-a587-4132863db3aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/127b4c5c-09d7-4536-a587-4132863db3aa%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBbuxyKW4e9oq\_PmD2y\_nZ%2BNom%2BHkjsFBjhSuyUUcCSFQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBbuxyKW4e9oq_PmD2y_nZ%2BNom%2BHkjsFBjhSuyUUcCSFQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![sunayana\_choudhary](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sunayana\_choudhary](https://discuss.elastic.co/u/sunayana_choudhary)\ **Post date:** [February 12, 2014, 6:30am UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/5 "2014-02-12T06:30:39Z")
Thanks Brin, your answer solved my problem. Thanks Ivan to you too, I am having 5 shards, idf is getting calculated on the maxdocs present in that shard. Doesn't that leads to misleading idf? On Tuesday, 11 February 2014 20:10:52 UTC+5:30, sunayana choudhary wrote: > Hi all, > > I have been analysing Elasticsearch results with explain:true condition, I > am not able to understand what technique has been applied to calculate > idf. I went through the lucene scoring formula i.e. > > idf(t) = 1+log(NumDocs/Doc frequency+1) > > Does not matches my results. > Following is explanation for one of the results returned. > \_explanation: { > value: 5.8878393 > description: weight(city:chicago in 1) [PerFieldSimilarity], > result of: > details: [ > { > value: 5.8878393 > description: score(doc=1,freq=1.0 = termFreq=1.0 ), > product of: > details: [ > { > value: 0.99999994 > description: queryWeight, product of: > details: [ > { > value: 5.88784 > description: idf(docFreq=2, maxDocs=398) > } > { > value: 0.16984157 > description: queryNorm > } > ] > } > { > value: 5.88784 > description: fieldWeight in 1, product of: > details: [ > { > value: 1 > description: tf(freq=1.0), with freq of: > details: [ > { > value: 1 > description: termFreq=1.0 > } > ] > } > { > value: 5.88784 > description: idf(docFreq=2, maxDocs=398) > } > { > value: 1 > description: fieldNorm(doc=1) > } > ] > } > ] > } > ] > } > > } > > Thanks in advance.. 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dc8907ed-59c5-4432-9472-b596e9e3a4ea%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dc8907ed-59c5-4432-9472-b596e9e3a4ea%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:50am UTC](https://discuss.elastic.co/t/computing-idf-in-elasticsearch/15731/6 "2017-07-06T01:50:50Z")