# Easiest way to get the IDF of a term from an ES cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [September 3, 2012, 9:54pm UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920 "2012-09-03T21:54:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Matt\_Luongo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_luongo/32/2735_2.png) [@Matt\_Luongo](https://discuss.elastic.co/u/Matt_Luongo)\ **Post date:** [September 3, 2012, 9:54pm UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/1 "2012-09-03T21:54:47Z")
Hi folks, I'm trying to retrieve the IDF of a few particular terms from an elasticsearch cluster. Though I have plenty of Java experience, I'm only somewhat familiar with Lucene and know nothing about ES internals. What would be the path of least resistance here? Thanks in advance! - Matt -- ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [September 5, 2012, 4:50am UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/2 "2012-09-05T04:50:42Z")
Since IDF is neither a document nor a field of a document, the only place to generate a dynamic number is in a facet. I would look into what term facet or terms\_stat or statistical facet can do and then register a native script which roots around in Java API to find the right value to return instead of the usual values it returns. I think I saw somewhere where you could get the right Lucene objects for asking for the IDF. If you can get the Lucene IndexSearcher you can call docFreq(term). Will that help? After all the claim is that idf = log(numDocs/(docFreq+1)) + 1 Since any of these facets is over all docs in the query, I guess you'd have to do this facet for one document (any document), but not all documents, since IDF is for the entire index for a term. Keep in mind that you can send params into a script, so maybe: "statistical" : { "script" : "someCraxyBitOfJava", "params" : { "idfForTerm" : "foo" } } I think that is 3 round pegs in 2.5 square holes, but I think it might work. 🙂 -Paul -- ---
**Author:** ![Matt\_Luongo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_luongo/32/2735_2.png) [@Matt\_Luongo](https://discuss.elastic.co/u/Matt_Luongo)\ **Post date:** [September 6, 2012, 8:19pm UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/3 "2012-09-06T20:19:39Z")
Thanks Paul! docFreq() would definitely help. I'll give this one a shot soon and report back. -- Matt Luongo Software Developer > **[Matt Luongo on about.me](https://about.me/luongo)** > > I use about.me to show people what matters most to me. On Wed, Sep 5, 2012 at 12:50 AM, P.Hill [parehill1@gmail.com](mailto:parehill1@gmail.com) wrote: > Since IDF is neither a document nor a field of a document, the only place > to generate a dynamic number is in a facet. > I would look into what term facet or terms\_stat or statistical facet can > do and then register a native script which roots around in Java API to find > the right value to return instead of the usual values it returns. > > I think I saw somewhere where you could get the right Lucene objects for > asking for the IDF. > If you can get the Lucene IndexSearcher you can call docFreq(term). Will > that help? After all the claim is that > > idf = log(numDocs/(docFreq+1)) + 1 > > Since any of these facets is over all docs in the query, I guess you'd > have to do this facet for one document (any document), but not all > documents, since IDF is for the entire index for a term. > > Keep in mind that you can send params into a script, so maybe: > > "statistical" : { > "script" : "someCraxyBitOfJava", > "params" : { > "idfForTerm" : "foo" > } > } > > I think that is 3 round pegs in 2.5 square holes, but I think it might > work. 🙂 > > -Paul > > -- -- ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [September 7, 2012, 12:17am UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/4 "2012-09-07T00:17:50Z")
But after I wrote that I thought I also recall someone mentioning a plug-in that could help with such index-level information. -Paul On 9/6/2012 1:19 PM, Matt Luongo wrote: > Thanks Paul! docFreq() would definitely help. I'll give this one a > shot soon and report back. -- ---
**Author:** ![Matt\_Luongo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_luongo/32/2735_2.png) [@Matt\_Luongo](https://discuss.elastic.co/u/Matt_Luongo)\ **Post date:** [September 7, 2012, 12:44am UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/5 "2012-09-07T00:44:16Z")
Maybe [GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene](https://github.com/jprante/elasticsearch-skywalker)? It looks like it can return high frequency terms, but not docFreq for a particular term. -- Matt Luongo Software Developer > **[Matt Luongo on about.me](https://about.me/luongo)** > > I use about.me to show people what matters most to me. On Thu, Sep 6, 2012 at 8:18 PM, P. Hill [parehill1@gmail.com](mailto:parehill1@gmail.com) wrote: > But after I wrote that I thought I also recall someone mentioning a > plug-in that could help with such index-level information. > > -Paul > > On 9/6/2012 1:19 PM, Matt Luongo wrote: > > > Thanks Paul! docFreq() would definitely help. I'll give this one a shot > > soon and report back. > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:13am UTC](https://discuss.elastic.co/t/easiest-way-to-get-the-idf-of-a-term-from-an-es-cluster/8920/6 "2017-07-06T03:13:51Z")